[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:06:49 dist-test-slave-64fx krb5kdc[4926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:06:49 dist-test-slave-64fx krb5kdc[4926](info): set up 2 sockets
Sep 25 15:06:49 dist-test-slave-64fx krb5kdc[4926](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 15:06:49.999136 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.009s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:06:50 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:06:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:06:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:06:50.161751 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41341
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:50.474284 4939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:50.474586 4939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:50.474752 4939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:50.482126 4939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:06:50.482257 4939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:50.482367 4939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:50.482457 4939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:06:50.482542 4939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:06:50.491760 4939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:50.493017 4939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:50.494349 4939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:50.507099 4945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:50.507169 4944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:50.507141 4947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:50.507551 4939 server_base.cc:1034] running on GCE node
I20260925 15:06:50.508322 4939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:50.509646 4939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:50.510844 4939 hybrid_clock.cc:648] HybridClock initialized: now 1790348810510791 us; error 45 us; skew 500 ppm
I20260925 15:06:50.511185 4939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:50 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348810, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:50.514071 4939 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:06:50.515496 4939 webserver.cc:460] Webserver started at http://127.4.205.254:44837/ using document root <none> and password file <none>
I20260925 15:06:50.516142 4939 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:50.516275 4939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:50.516554 4939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:50.519426 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "7671f4c3ca4c40229048f7fa23772c25"
format_stamp: "Formatted at 2026-09-25 15:06:50 on dist-test-slave-64fx"
server_key: "fdccd1a9667c834da5c0e726732f77cb"
server_key_iv: "9e20d2dedd2cbf0c15a3ddd570187696"
server_key_version: "encryptionkey@0"
I20260925 15:06:50.520084 4939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "7671f4c3ca4c40229048f7fa23772c25"
format_stamp: "Formatted at 2026-09-25 15:06:50 on dist-test-slave-64fx"
server_key: "fdccd1a9667c834da5c0e726732f77cb"
server_key_iv: "9e20d2dedd2cbf0c15a3ddd570187696"
server_key_version: "encryptionkey@0"
I20260925 15:06:50.525313 4939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:06:50.528317 4954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:50.529482 4939 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:06:50.529690 4939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "7671f4c3ca4c40229048f7fa23772c25"
format_stamp: "Formatted at 2026-09-25 15:06:50 on dist-test-slave-64fx"
server_key: "fdccd1a9667c834da5c0e726732f77cb"
server_key_iv: "9e20d2dedd2cbf0c15a3ddd570187696"
server_key_version: "encryptionkey@0"
I20260925 15:06:50.529870 4939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:50.542649 4939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:50.546201 4939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:50.546454 4939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:50.564723 4939 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41341
I20260925 15:06:50.564743 5005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41341 every 8 connection(s)
I20260925 15:06:50.565961 4939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:06:50.569293 5006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:50.576201 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 4939
I20260925 15:06:50.576515 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:06:50.578402 4919 external_mini_cluster.cc:1428] Setting key d7e6fb834c56a9678feacd0c59055de1
I20260925 15:06:50.579238 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25: Bootstrap starting.
I20260925 15:06:50.582566 5006 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:50.583631 5006 log.cc:824] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:50.586524 5006 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25: No bootstrap required, opened a new log
I20260925 15:06:50.594465 5006 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } }
I20260925 15:06:50.594794 5006 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:50.594906 5006 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7671f4c3ca4c40229048f7fa23772c25, State: Initialized, Role: FOLLOWER
I20260925 15:06:50.595471 5006 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } }
I20260925 15:06:50.595677 5006 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:06:50.595813 5006 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:06:50.595980 5006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:50.598184 5006 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } }
I20260925 15:06:50.598639 5006 leader_election.cc:302] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7671f4c3ca4c40229048f7fa23772c25; no voters:
I20260925 15:06:50.599327 5006 leader_election.cc:288] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:06:50.599411 5011 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:50.601428 5011 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [term 1 LEADER]: Becoming Leader. State: Replica: 7671f4c3ca4c40229048f7fa23772c25, State: Running, Role: LEADER
I20260925 15:06:50.601541 5006 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:06:50.602021 5011 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } }
I20260925 15:06:50.606523 5013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7671f4c3ca4c40229048f7fa23772c25. Latest consensus state: current_term: 1 leader_uuid: "7671f4c3ca4c40229048f7fa23772c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } } }
I20260925 15:06:50.606870 5013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [sys.catalog]: This master's current role is: LEADER
Sep 25 15:06:50 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348810, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:50.614969 5018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:06:50.615064 5012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7671f4c3ca4c40229048f7fa23772c25" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7671f4c3ca4c40229048f7fa23772c25" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41341 } } }
I20260925 15:06:50.615345 5012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25 [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:50.622104 5018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:06:50.624914 5008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:50.581981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46124 (local address 127.4.205.254:41341)
0925 15:06:50.582889 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:50.582908 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:50.584114 (+ 1206us) server_negotiation.cc:407] Connection header received
0925 15:06:50.594037 (+ 9923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:50.594057 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:50.594387 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:50.594767 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:50.597651 (+ 2884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:50.599148 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:50.601207 (+ 2059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:50.602194 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:50.602402 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:50.611176 (+ 8774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:50.611191 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:50.611215 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:50.611276 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:50.617442 (+ 6166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:50.618715 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:50.618727 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:50.618738 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:50.619042 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:50.619591 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:50.619598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:50.619604 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:50.620356 (+ 752us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:50.620743 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:50.623089 (+ 2346us) server_negotiation.cc:299] Negotiation successful
0925 15:06:50.623590 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":683,"thread_start_us":417,"threads_started":1}
I20260925 15:06:50.633262 5018 catalog_manager.cc:1347] Generated new cluster ID: bbd6060835a94806bc226a40f7c2e3f1
I20260925 15:06:50.633359 5018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:06:50.651839 5018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:06:50.653703 5018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:06:50.663488 5018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7671f4c3ca4c40229048f7fa23772c25: Generated new TSK 0
I20260925 15:06:50.664513 5018 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:06:50.705245 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:51.022853 5032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:51.023156 5032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:51.023337 5032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:51.030190 5032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:51.030328 5032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:51.030503 5032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:06:51.039613 5032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:51.041143 5032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:51.042464 5032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:51.056291 5037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.056304 5038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.056308 5040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:51.057224 5032 server_base.cc:1034] running on GCE node
I20260925 15:06:51.057819 5032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:51.058638 5032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:51.059834 5032 hybrid_clock.cc:648] HybridClock initialized: now 1790348811059799 us; error 45 us; skew 500 ppm
I20260925 15:06:51.060099 5032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:51 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:51.062964 5032 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:06:51.064320 5032 webserver.cc:460] Webserver started at http://127.4.205.193:35665/ using document root <none> and password file <none>
I20260925 15:06:51.064932 5032 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:51.065068 5032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:51.065357 5032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:51.067871 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "3cdb78b35f994a33bf06eb5f0d0d2208"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "8945c0c54b1a54093bfcc0d92fd8196a"
server_key_iv: "aeb565b58a3bdf2663991a11071c4bbd"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.068580 5032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "3cdb78b35f994a33bf06eb5f0d0d2208"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "8945c0c54b1a54093bfcc0d92fd8196a"
server_key_iv: "aeb565b58a3bdf2663991a11071c4bbd"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.073166 5032 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:06:51.076256 5047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.077401 5032 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:06:51.077586 5032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "3cdb78b35f994a33bf06eb5f0d0d2208"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "8945c0c54b1a54093bfcc0d92fd8196a"
server_key_iv: "aeb565b58a3bdf2663991a11071c4bbd"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.077764 5032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:51.096698 5032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:51.100237 5032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:51.100502 5032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:51.101488 5032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:51.102859 5032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:51.102953 5032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.103080 5032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:51.103159 5032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.137274 5032 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:41817
I20260925 15:06:51.137305 5159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:41817 every 8 connection(s)
I20260925 15:06:51.138530 5032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:06:51.140020 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5032
I20260925 15:06:51.140604 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:06:51.141332 4919 external_mini_cluster.cc:1428] Setting key a36feaef61307e2311d6eaf305f23340
Sep 25 15:06:51 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:06:51.158849 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:51.140847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57841 (local address 127.4.205.254:41341)
0925 15:06:51.141366 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:51.141373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:51.143067 (+ 1694us) server_negotiation.cc:407] Connection header received
0925 15:06:51.144815 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:51.144818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:51.144878 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:51.144961 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:51.147013 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.147697 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.149187 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.149602 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.149736 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:51.153004 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:51.153011 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:51.153024 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:51.153054 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:51.155954 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:51.156683 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:51.156689 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:51.156694 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:51.156861 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:51.157226 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:51.157247 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:51.157249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:51.157506 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:51.157692 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:51.158439 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 15:06:51.158591 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":262,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
I20260925 15:06:51.160221 5163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:51.141524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41341 (local address 127.4.205.193:57841)
0925 15:06:51.142343 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:51.142396 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:51.144544 (+ 2148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:51.145225 (+ 681us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:51.145238 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:51.146058 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:51.146840 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:51.146866 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.147857 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.147860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:51.149026 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:51.149038 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.149832 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.149841 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:51.150064 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:51.150708 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:51.150734 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:51.152802 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:51.156098 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:51.156108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:51.156125 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:51.156534 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:51.156961 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:51.156964 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:51.156966 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:51.157135 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:51.157720 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:51.157726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:51.158104 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 15:06:51.158356 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 15:06:51.158847 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":267,"threads_started":1}
I20260925 15:06:51.161835 5160 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41341
I20260925 15:06:51.162127 5160 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:51.162838 5160 heartbeater.cc:511] Master 127.4.205.254:41341 requested a full tablet report, sending...
I20260925 15:06:51.165041 4971 ts_manager.cc:194] Registered new tserver with Master: 3cdb78b35f994a33bf06eb5f0d0d2208 (127.4.205.193:41817)
I20260925 15:06:51.166738 4971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:57841
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:06:51.180905 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:51.489452 5166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:51.489754 5166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:51.489924 5166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:51.496598 5166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:51.496737 5166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:51.496920 5166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:06:51.506016 5166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:51.507184 5166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:51.508438 5166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:51.520917 5172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.521164 5174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.520956 5171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:51.521973 5166 server_base.cc:1034] running on GCE node
I20260925 15:06:51.522506 5166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:51.523731 5166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:51.524977 5166 hybrid_clock.cc:648] HybridClock initialized: now 1790348811524925 us; error 48 us; skew 500 ppm
I20260925 15:06:51.525341 5166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:51 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:51.531893 5166 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:06:51.533380 5166 webserver.cc:460] Webserver started at http://127.4.205.194:40683/ using document root <none> and password file <none>
I20260925 15:06:51.534119 5166 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:51.534255 5166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:51.534623 5166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:51.537168 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "945aa77371bb4161a2a17ddc7090dd0b"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "d4708e2aca98741f92204792291167a0"
server_key_iv: "cf66041e2486f8929fc5024c8cba2107"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.537880 5166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "945aa77371bb4161a2a17ddc7090dd0b"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "d4708e2aca98741f92204792291167a0"
server_key_iv: "cf66041e2486f8929fc5024c8cba2107"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.542428 5166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:06:51.545677 5181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.546819 5166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:06:51.546969 5166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "945aa77371bb4161a2a17ddc7090dd0b"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "d4708e2aca98741f92204792291167a0"
server_key_iv: "cf66041e2486f8929fc5024c8cba2107"
server_key_version: "encryptionkey@0"
I20260925 15:06:51.547147 5166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:51.563697 5166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:51.567606 5166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:51.567870 5166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:51.568660 5166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:51.569818 5166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:51.569882 5166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.570030 5166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:51.570104 5166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:51.607020 5166 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:34423
I20260925 15:06:51.607050 5293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:34423 every 8 connection(s)
I20260925 15:06:51.608217 5166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:06:51.614789 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5166
I20260925 15:06:51.614984 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:06:51.615382 4919 external_mini_cluster.cc:1428] Setting key fe5aa400e0b25e35b80a6db8033b4d8a
Sep 25 15:06:51 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:51.628295 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:51.610492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:36895 (local address 127.4.205.254:41341)
0925 15:06:51.610641 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:51.610646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:51.611679 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 15:06:51.613346 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:51.613351 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:51.613439 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:51.613545 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:51.615017 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.615820 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.617415 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.618233 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.618358 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:51.621854 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:51.621859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:51.621862 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:51.621890 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:51.625247 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:51.625963 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:51.625971 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:51.625977 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:51.626128 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:51.626502 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:51.626510 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:51.626517 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:51.626933 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:51.627083 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:51.627674 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:06:51.627932 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:06:51.628939 5296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:51.610897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41341 (local address 127.4.205.194:36895)
0925 15:06:51.611537 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:51.611571 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:51.613070 (+ 1499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:51.613792 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:51.613800 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:51.614283 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:51.614815 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:51.614831 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.615966 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.615971 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:51.617292 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:51.617303 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:51.618434 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:51.618441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:51.618647 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:51.619295 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:51.619318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:51.621670 (+ 2352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:51.625375 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:51.625381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:51.625392 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:51.625855 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:51.626231 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:51.626234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:51.626236 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:51.626382 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:51.627047 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:51.627052 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:51.627329 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 15:06:51.627528 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:06:51.627937 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":238,"threads_started":1}
I20260925 15:06:51.630299 5294 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41341
I20260925 15:06:51.630610 5294 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:51.631482 5294 heartbeater.cc:511] Master 127.4.205.254:41341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:06:51.632967 4971 ts_manager.cc:194] Registered new tserver with Master: 945aa77371bb4161a2a17ddc7090dd0b (127.4.205.194:34423)
I20260925 15:06:51.633939 4971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:36895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:06:51.656445 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:51.957152 5299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:51.957470 5299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:51.957685 5299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:51.964305 5299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:51.964440 5299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:51.964636 5299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:06:51.973831 5299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41341
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:51.974988 5299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:51.976214 5299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:51.989068 5304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.989003 5305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:51.989045 5307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:51.989655 5299 server_base.cc:1034] running on GCE node
I20260925 15:06:51.990276 5299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:51.991117 5299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:51.992306 5299 hybrid_clock.cc:648] HybridClock initialized: now 1790348811992252 us; error 60 us; skew 500 ppm
I20260925 15:06:51.992599 5299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:51 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:51.995555 5299 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:06:51.996896 5299 webserver.cc:460] Webserver started at http://127.4.205.195:39523/ using document root <none> and password file <none>
I20260925 15:06:51.997689 5299 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:51.997826 5299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:51.998137 5299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:52.000907 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "e304112f672f480e82446f2ea111cf8c"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "806cc930a86f3aeb9c121be1fc0eafc2"
server_key_iv: "5890233c0b95bf03bea48a90dffb478c"
server_key_version: "encryptionkey@0"
I20260925 15:06:52.001773 5299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "e304112f672f480e82446f2ea111cf8c"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "806cc930a86f3aeb9c121be1fc0eafc2"
server_key_iv: "5890233c0b95bf03bea48a90dffb478c"
server_key_version: "encryptionkey@0"
I20260925 15:06:52.007160 5299 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:06:52.010586 5314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:52.011739 5299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:06:52.011901 5299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "e304112f672f480e82446f2ea111cf8c"
format_stamp: "Formatted at 2026-09-25 15:06:51 on dist-test-slave-64fx"
server_key: "806cc930a86f3aeb9c121be1fc0eafc2"
server_key_iv: "5890233c0b95bf03bea48a90dffb478c"
server_key_version: "encryptionkey@0"
I20260925 15:06:52.012072 5299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:52.023468 5299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:52.027413 5299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:52.027680 5299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:52.028517 5299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:52.029775 5299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:52.029873 5299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:52.030009 5299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:52.030076 5299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:52.064174 5299 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:35337
I20260925 15:06:52.064211 5426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:35337 every 8 connection(s)
I20260925 15:06:52.065322 5299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:06:52.069053 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5299
I20260925 15:06:52.069278 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:06:52.069751 4919 external_mini_cluster.cc:1428] Setting key aa46e31a824510c1b63831cbd62485e8
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348811, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:52.083683 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.067646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:55553 (local address 127.4.205.254:41341)
0925 15:06:52.067806 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.067810 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.068849 (+ 1039us) server_negotiation.cc:407] Connection header received
0925 15:06:52.070698 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.070702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.070771 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.070888 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.072454 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.073410 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.075255 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.075627 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.075714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.078646 (+ 2932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.078650 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.078653 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.078681 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.080874 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.081636 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.081640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.081642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.081782 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.082201 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.082204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.082206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.082463 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.082559 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.083176 (+ 617us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.083338 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:06:52.084625 5429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.068101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41341 (local address 127.4.205.195:55553)
0925 15:06:52.068711 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:52.068752 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:52.070438 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:52.071145 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:52.071157 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:52.071657 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:52.072290 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:52.072306 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.073973 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.073978 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:52.075145 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:52.075156 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.075794 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.075798 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:52.075970 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.076522 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:52.076541 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:52.078487 (+ 1946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:52.080980 (+ 2493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:52.080985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:52.080995 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:52.081470 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:52.081875 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:52.081879 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:52.081881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:52.082097 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:52.082572 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:52.082582 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:52.082892 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 15:06:52.083091 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:06:52.083496 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":223,"threads_started":1}
I20260925 15:06:52.085877 5427 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41341
I20260925 15:06:52.086171 5427 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:52.086778 5427 heartbeater.cc:511] Master 127.4.205.254:41341 requested a full tablet report, sending...
I20260925 15:06:52.088089 4971 ts_manager.cc:194] Registered new tserver with Master: e304112f672f480e82446f2ea111cf8c (127.4.205.195:35337)
I20260925 15:06:52.088837 4971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:55553
I20260925 15:06:52.097092 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:06:52.137912 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.123989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43050 (local address 127.4.205.193:41817)
0925 15:06:52.124761 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.124768 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.124794 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:06:52.124930 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.124936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.125149 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.125293 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.126632 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.127617 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.128730 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.129202 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.129309 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.132754 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.132763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.132775 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.132802 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.135547 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.136026 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.136031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.136033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.136136 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.136434 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.136437 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.136438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.136789 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.137067 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.137258 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.137465 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":540,"threads_started":1}
W20260925 15:06:52.139189 5075 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43050
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 15:06:52.170244 5160 heartbeater.cc:503] Master 127.4.205.254:41341 was elected leader, sending a full tablet report...
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:06:52.178179 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.165326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43056 (local address 127.4.205.193:41817)
0925 15:06:52.165490 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.165494 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.165849 (+ 355us) server_negotiation.cc:407] Connection header received
0925 15:06:52.165984 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.165988 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.166095 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.166184 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.167400 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.168099 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.169248 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.169674 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.169764 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.173627 (+ 3863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.173631 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.173635 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.173664 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.175822 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.176378 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.176389 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.176392 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.176492 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.176815 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.176821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.176825 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.177073 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.177301 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.177667 (+ 366us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.177893 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:06:52.179378 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5032
I20260925 15:06:52.190946 5434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.180258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43068 (local address 127.4.205.193:41817)
0925 15:06:52.180425 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.180428 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.180628 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:06:52.180726 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.180729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.180773 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.180839 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.182027 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.182759 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.184372 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.184737 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.184856 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.186701 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.186705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.186708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.186733 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.188993 (+ 2260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.189479 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.189482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.189484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.189651 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.189935 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.189937 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.189939 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.190159 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.190333 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.190578 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.190706 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:06:52.191567 5155 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:52.272650 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5032
I20260925 15:06:52.287966 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5166
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:06:52.302740 5441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.288789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.4.205.194:34423)
0925 15:06:52.289298 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.289305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.289328 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:06:52.289389 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.289395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.289660 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.289801 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.290828 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.291710 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.292815 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.293344 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.293442 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.297434 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.297444 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.297456 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.297484 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.299940 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.300578 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.300586 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.300588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.300712 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.301126 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.301132 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.301135 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.301482 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.301813 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.302135 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.302403 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":274,"threads_started":1}
I20260925 15:06:52.304060 5289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:52.382161 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5166
I20260925 15:06:52.404894 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5299
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[4926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:06:52.419574 5443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:52.405880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.4.205.195:35337)
0925 15:06:52.406458 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:52.406466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:52.406490 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:06:52.406543 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:52.406549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:52.406745 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:52.406864 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:52.408019 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.408977 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.410149 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:52.410636 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:52.410723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:52.414496 (+ 3773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:52.414505 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:52.414517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:52.414545 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:52.417150 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.417883 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.417890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.417892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.418008 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:52.418306 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:52.418309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:52.418311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:52.418601 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:52.418812 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:52.419069 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:06:52.419262 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":452,"thread_start_us":289,"threads_started":1}
I20260925 15:06:52.420852 5422 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:52.499114 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5299
I20260925 15:06:52.514722 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 4939
I20260925 15:06:52.515825 5001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:52.612367 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 4939
2026-09-25T15:06:52Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3665 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[5448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[5448](info): set up 2 sockets
Sep 25 15:06:52 dist-test-slave-64fx krb5kdc[5448](info): commencing operation
krb5kdc: starting...
W20260925 15:06:53.637408 4919 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:06:53 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:06:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:06:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:06:53.798827 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:32921
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:32921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:54.095736 5461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:54.096000 5461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:54.096082 5461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:54.102824 5461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:06:54.102916 5461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:54.102959 5461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:54.103005 5461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:06:54.103048 5461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:06:54.111963 5461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:32921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:32921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:54.113027 5461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:54.114209 5461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:54.124935 5466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:54.124990 5467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:54.124972 5469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:54.125779 5461 server_base.cc:1034] running on GCE node
I20260925 15:06:54.126432 5461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:54.127612 5461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:54.128810 5461 hybrid_clock.cc:648] HybridClock initialized: now 1790348814128778 us; error 42 us; skew 500 ppm
I20260925 15:06:54.129107 5461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:54 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:54.131784 5461 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:06:54.133023 5461 webserver.cc:460] Webserver started at http://127.4.205.254:32809/ using document root <none> and password file <none>
I20260925 15:06:54.133687 5461 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:54.133785 5461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:54.134016 5461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:54.136533 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d107f0c7555d4c10a5bd2377f38266f7"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "3a6edb5858c8a9ffc852cbd5e89d8c17"
server_key_iv: "272ee02a5c8a7a064b75431a0c5a1d95"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.137221 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d107f0c7555d4c10a5bd2377f38266f7"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "3a6edb5858c8a9ffc852cbd5e89d8c17"
server_key_iv: "272ee02a5c8a7a064b75431a0c5a1d95"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.141544 5461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:06:54.144578 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:54.145751 5461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260925 15:06:54.145902 5461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d107f0c7555d4c10a5bd2377f38266f7"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "3a6edb5858c8a9ffc852cbd5e89d8c17"
server_key_iv: "272ee02a5c8a7a064b75431a0c5a1d95"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.146061 5461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:54.167500 5461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:54.171096 5461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:54.171371 5461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:54.188325 5461 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:32921
I20260925 15:06:54.188359 5527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:32921 every 8 connection(s)
I20260925 15:06:54.189595 5461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:06:54.192854 5528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:54.193275 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5461
I20260925 15:06:54.193507 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:06:54.194343 4919 external_mini_cluster.cc:1428] Setting key 1044f17272e283d5e278e1ffc2b7a63d
I20260925 15:06:54.205854 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7: Bootstrap starting.
Sep 25 15:06:54 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:54.209201 5528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:54.210453 5528 log.cc:824] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:54.213366 5528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7: No bootstrap required, opened a new log
I20260925 15:06:54.216714 5531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:54.196252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37516 (local address 127.4.205.254:32921)
0925 15:06:54.196934 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:54.196943 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:54.196983 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:06:54.198740 (+ 1757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:54.198758 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:54.199117 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:54.199399 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:54.200892 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.202219 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.203881 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.204489 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.204625 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:54.208866 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:54.208887 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:54.208904 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:54.208940 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:54.212510 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:54.213083 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:54.213091 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:54.213098 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:54.213294 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:54.213739 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:54.213744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:54.213747 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:54.214395 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:54.214687 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:54.215014 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 15:06:54.215435 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":321,"threads_started":1}
I20260925 15:06:54.222198 5528 raft_consensus.cc:348] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } }
I20260925 15:06:54.222584 5528 raft_consensus.cc:374] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:54.222677 5528 raft_consensus.cc:729] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d107f0c7555d4c10a5bd2377f38266f7, State: Initialized, Role: FOLLOWER
I20260925 15:06:54.223242 5528 consensus_queue.cc:260] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } }
I20260925 15:06:54.223474 5528 raft_consensus.cc:388] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:06:54.223599 5528 raft_consensus.cc:482] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:06:54.223765 5528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:54.226287 5528 raft_consensus.cc:504] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } }
I20260925 15:06:54.226723 5528 leader_election.cc:302] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d107f0c7555d4c10a5bd2377f38266f7; no voters:
I20260925 15:06:54.227291 5528 leader_election.cc:288] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:06:54.227408 5533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:54.228178 5533 raft_consensus.cc:686] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [term 1 LEADER]: Becoming Leader. State: Replica: d107f0c7555d4c10a5bd2377f38266f7, State: Running, Role: LEADER
I20260925 15:06:54.228773 5533 consensus_queue.cc:237] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } }
I20260925 15:06:54.229254 5528 sys_catalog.cc:564] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:06:54.231325 5535 sys_catalog.cc:455] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d107f0c7555d4c10a5bd2377f38266f7. Latest consensus state: current_term: 1 leader_uuid: "d107f0c7555d4c10a5bd2377f38266f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } } }
I20260925 15:06:54.231355 5534 sys_catalog.cc:455] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d107f0c7555d4c10a5bd2377f38266f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d107f0c7555d4c10a5bd2377f38266f7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 32921 } } }
I20260925 15:06:54.231736 5535 sys_catalog.cc:458] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:54.231736 5534 sys_catalog.cc:458] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:54.233294 5538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:06:54.238912 5538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:06:54.248696 5538 catalog_manager.cc:1347] Generated new cluster ID: 08d8915a47a64cadb85251f4ff38f550
I20260925 15:06:54.248799 5538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:06:54.260756 5538 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:06:54.262552 5538 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:06:54.271106 5538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d107f0c7555d4c10a5bd2377f38266f7: Generated new TSK 0
I20260925 15:06:54.272028 5538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:06:54.314666 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:54.627441 5554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:54.627672 5554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:54.627756 5554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:54.634519 5554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:54.634653 5554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:54.634827 5554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:06:54.643769 5554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:54.644896 5554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:54.646148 5554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:54.658226 5559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:54.658242 5560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:54.658282 5562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:54.659336 5554 server_base.cc:1034] running on GCE node
I20260925 15:06:54.659852 5554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:54.660660 5554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:54.661860 5554 hybrid_clock.cc:648] HybridClock initialized: now 1790348814661821 us; error 48 us; skew 500 ppm
I20260925 15:06:54.662139 5554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:54 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:54.664966 5554 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:06:54.666323 5554 webserver.cc:460] Webserver started at http://127.4.205.193:46529/ using document root <none> and password file <none>
I20260925 15:06:54.666916 5554 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:54.667047 5554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:54.667327 5554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:54.670022 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "8d43d5d5bc56452a89d3662f4107e10f"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "76f569841ba38b25d43a20c25b7cc76c"
server_key_iv: "4a0ec004bf7fb3b63cee0bd3c7684a30"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.670787 5554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "8d43d5d5bc56452a89d3662f4107e10f"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "76f569841ba38b25d43a20c25b7cc76c"
server_key_iv: "4a0ec004bf7fb3b63cee0bd3c7684a30"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.675417 5554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:06:54.678462 5569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:54.679553 5554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:06:54.679701 5554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "8d43d5d5bc56452a89d3662f4107e10f"
format_stamp: "Formatted at 2026-09-25 15:06:54 on dist-test-slave-64fx"
server_key: "76f569841ba38b25d43a20c25b7cc76c"
server_key_iv: "4a0ec004bf7fb3b63cee0bd3c7684a30"
server_key_version: "encryptionkey@0"
I20260925 15:06:54.679874 5554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:54.700670 5554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:54.704439 5554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:54.704784 5554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:54.705745 5554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:54.706984 5554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:54.707088 5554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:54.707223 5554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:54.707289 5554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:54.738554 5554 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:38115
I20260925 15:06:54.738621 5681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:38115 every 8 connection(s)
I20260925 15:06:54.739669 5554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:06:54.748932 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5554
I20260925 15:06:54.749123 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:06:54.749889 4919 external_mini_cluster.cc:1428] Setting key 5cdf43ae3189a10ffe100ae87156ed46
Sep 25 15:06:54 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348814, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:54.759171 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:54.741950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:38383 (local address 127.4.205.254:32921)
0925 15:06:54.742390 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:54.742394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:54.743144 (+ 750us) server_negotiation.cc:407] Connection header received
0925 15:06:54.744890 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:54.744895 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:54.744981 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:54.745104 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:54.746520 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.747218 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.748653 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.749102 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.749251 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:54.752641 (+ 3390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:54.752650 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:54.752659 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:54.752701 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:54.756199 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:54.756971 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:54.756977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:54.756981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:54.757129 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:54.757504 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:54.757512 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:54.757517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:54.757844 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:54.757974 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:54.758786 (+ 812us) server_negotiation.cc:299] Negotiation successful
0925 15:06:54.758931 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":249,"threads_started":1}
I20260925 15:06:54.760699 5685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:54.742372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:32921 (local address 127.4.205.193:38383)
0925 15:06:54.742997 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:54.743035 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:54.744642 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:54.745313 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:54.745321 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:54.745839 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:54.746353 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:54.746374 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.747367 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.747371 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:54.748521 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:54.748537 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:54.749316 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:54.749324 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:54.749681 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:54.750382 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:54.750416 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:54.752442 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:54.756371 (+ 3929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:54.756382 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:54.756397 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:54.756857 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:54.757233 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:54.757236 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:54.757238 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:54.757393 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:54.757979 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:54.757990 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:54.758432 (+ 442us) client_negotiation.cc:769] Sending connection context
0925 15:06:54.758710 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 15:06:54.759263 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":249,"threads_started":1}
I20260925 15:06:54.762176 5682 heartbeater.cc:347] Connected to a master server at 127.4.205.254:32921
I20260925 15:06:54.762470 5682 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:54.763098 5682 heartbeater.cc:511] Master 127.4.205.254:32921 requested a full tablet report, sending...
I20260925 15:06:54.765054 5493 ts_manager.cc:194] Registered new tserver with Master: 8d43d5d5bc56452a89d3662f4107e10f (127.4.205.193:38115)
I20260925 15:06:54.766446 5493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:38383
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:06:54.789053 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:55.087446 5688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:55.087683 5688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:55.087769 5688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:55.094341 5688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:55.094432 5688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:55.094539 5688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:06:55.103493 5688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:55.104553 5688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:55.105809 5688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:55.117764 5693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:55.117709 5696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:55.117794 5694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:55.119005 5688 server_base.cc:1034] running on GCE node
I20260925 15:06:55.119470 5688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:55.120209 5688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:55.121413 5688 hybrid_clock.cc:648] HybridClock initialized: now 1790348815121369 us; error 36 us; skew 500 ppm
I20260925 15:06:55.121747 5688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:55.124699 5688 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:06:55.126041 5688 webserver.cc:460] Webserver started at http://127.4.205.194:43205/ using document root <none> and password file <none>
I20260925 15:06:55.126626 5688 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:55.126717 5688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:55.126945 5688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:55.129616 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "9aec845ea5ce49e4afb85e1887d788bb"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "97bc50c61b87d0eaec69c62eb7f3c188"
server_key_iv: "1cdc772822d7a035fd324a9989892a22"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.130329 5688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "9aec845ea5ce49e4afb85e1887d788bb"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "97bc50c61b87d0eaec69c62eb7f3c188"
server_key_iv: "1cdc772822d7a035fd324a9989892a22"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.134817 5688 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:06:55.137770 5703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.138919 5688 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:06:55.139052 5688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "9aec845ea5ce49e4afb85e1887d788bb"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "97bc50c61b87d0eaec69c62eb7f3c188"
server_key_iv: "1cdc772822d7a035fd324a9989892a22"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.139190 5688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:55.154307 5688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:55.158133 5688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:55.158339 5688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:55.159143 5688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:55.160250 5688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:55.160334 5688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.160414 5688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:55.160456 5688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.192173 5688 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:44655
I20260925 15:06:55.192191 5815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:44655 every 8 connection(s)
I20260925 15:06:55.193300 5688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:06:55.202318 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5688
I20260925 15:06:55.202505 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:06:55.202961 4919 external_mini_cluster.cc:1428] Setting key bd967aec31adfac0c643ec049dd9eba2
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:55.212615 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.195804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:33819 (local address 127.4.205.254:32921)
0925 15:06:55.195990 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.196007 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.197207 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 15:06:55.199008 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.199014 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.199128 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.199225 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.200463 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.201554 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.203199 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.203740 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.203875 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.207009 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.207015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.207021 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.207066 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.209960 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.210599 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.210606 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.210614 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.210730 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.211111 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.211117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.211121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.211353 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.211492 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.212119 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.212321 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:06:55.213490 5818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.196294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:32921 (local address 127.4.205.194:33819)
0925 15:06:55.197050 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.197100 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.198758 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.199400 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.199407 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.199836 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:55.200319 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.200333 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.201701 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.201705 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.203047 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.203061 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.203958 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.203965 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.204202 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.204835 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:55.204855 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:55.206807 (+ 1952us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:55.210089 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:55.210098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:55.210111 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:55.210470 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:55.210829 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:55.210835 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:55.210840 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:55.211006 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:55.211466 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:55.211473 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:55.211752 (+ 279us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.212009 (+ 257us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.212455 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":335,"threads_started":1}
I20260925 15:06:55.214999 5816 heartbeater.cc:347] Connected to a master server at 127.4.205.254:32921
I20260925 15:06:55.215289 5816 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:55.215946 5816 heartbeater.cc:511] Master 127.4.205.254:32921 requested a full tablet report, sending...
I20260925 15:06:55.217311 5493 ts_manager.cc:194] Registered new tserver with Master: 9aec845ea5ce49e4afb85e1887d788bb (127.4.205.194:44655)
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:06:55.218509 5493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:33819
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:06:55.238531 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:55.538348 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:55.538579 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:55.538744 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:55.545409 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:55.545518 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:55.545732 5821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:06:55.554760 5821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:32921
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:55.555846 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:55.557051 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:55.569156 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:55.569268 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:55.569244 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:55.569904 5821 server_base.cc:1034] running on GCE node
I20260925 15:06:55.570446 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:55.571314 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:55.572510 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790348815572460 us; error 47 us; skew 500 ppm
I20260925 15:06:55.572784 5821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:55.575683 5821 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:06:55.576797 5821 webserver.cc:460] Webserver started at http://127.4.205.195:37793/ using document root <none> and password file <none>
I20260925 15:06:55.577342 5821 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:55.577425 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:55.577713 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:55.580190 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "da9ec5cf44954f29ba3c1122d6a3de49"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "a139358d9aa24ac463c8cb7512e2119f"
server_key_iv: "afa39663228c59d110e0df1deee02e59"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.580834 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "da9ec5cf44954f29ba3c1122d6a3de49"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "a139358d9aa24ac463c8cb7512e2119f"
server_key_iv: "afa39663228c59d110e0df1deee02e59"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.585623 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:06:55.588605 5836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.589824 5821 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.003s
I20260925 15:06:55.589942 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "da9ec5cf44954f29ba3c1122d6a3de49"
format_stamp: "Formatted at 2026-09-25 15:06:55 on dist-test-slave-64fx"
server_key: "a139358d9aa24ac463c8cb7512e2119f"
server_key_iv: "afa39663228c59d110e0df1deee02e59"
server_key_version: "encryptionkey@0"
I20260925 15:06:55.590085 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:55.622857 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:55.626943 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:55.627206 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:55.628086 5821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:06:55.629339 5821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:55.629427 5821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.629529 5821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:55.629762 5821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:55.659336 5821 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:38253
I20260925 15:06:55.659410 5948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:38253 every 8 connection(s)
I20260925 15:06:55.660456 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:06:55.662513 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 5821
I20260925 15:06:55.662701 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:06:55.663152 4919 external_mini_cluster.cc:1428] Setting key 8b131fa7b08860ee49e2e15f38c83bb5
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:55.678525 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.662737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:52051 (local address 127.4.205.254:32921)
0925 15:06:55.662919 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.662926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.664513 (+ 1587us) server_negotiation.cc:407] Connection header received
0925 15:06:55.666562 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.666565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.666636 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.666712 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.667897 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.668544 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.669977 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.670364 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.670452 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.673353 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.673358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.673361 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.673390 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.675731 (+ 2341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.676357 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.676362 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.676365 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.676502 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.676897 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.676904 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.676909 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.677178 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.677342 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.677999 (+ 657us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.678202 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:06:55.679551 5951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.663220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:32921 (local address 127.4.205.195:52051)
0925 15:06:55.664382 (+ 1162us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.664428 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.666367 (+ 1939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.666887 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.666894 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.667322 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:55.667783 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.667797 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.668664 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.668667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.669874 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.669885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.670526 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.670530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.670710 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.671240 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:55.671262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:55.673167 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:55.675854 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:55.675860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:55.675869 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:55.676255 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:55.676620 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:55.676623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:55.676625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:55.676785 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:55.677316 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:55.677324 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:55.677684 (+ 360us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.677930 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.678391 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":897,"thread_start_us":714,"threads_started":1}
I20260925 15:06:55.680837 5949 heartbeater.cc:347] Connected to a master server at 127.4.205.254:32921
I20260925 15:06:55.681125 5949 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:55.682206 5949 heartbeater.cc:511] Master 127.4.205.254:32921 requested a full tablet report, sending...
I20260925 15:06:55.683447 5493 ts_manager.cc:194] Registered new tserver with Master: da9ec5cf44954f29ba3c1122d6a3de49 (127.4.205.195:38253)
I20260925 15:06:55.684276 5493 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:52051
I20260925 15:06:55.690078 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:06:55.707592 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.696347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37530 (local address 127.4.205.254:32921)
0925 15:06:55.696554 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.696563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.696948 (+ 385us) server_negotiation.cc:407] Connection header received
0925 15:06:55.697070 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.697073 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.697129 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.697207 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.698599 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.699292 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.700492 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.700910 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.700999 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.703053 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.703057 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.703060 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.703088 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.705542 (+ 2454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.706147 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.706150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.706152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.706280 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.706547 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.706550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.706552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.706805 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.706972 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.707208 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.707353 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:06:55.713539 5493 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:06:55.720163 5493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:06:55.743310 5961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.730597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:38253 (local address 127.0.0.1:47788)
0925 15:06:55.732134 (+ 1537us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.732172 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.732307 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.736000 (+ 3693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.736004 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.736026 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:55.736409 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.736421 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.737953 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.737958 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.740453 (+ 2495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.740465 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.742313 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.742319 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.742581 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.742633 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.742741 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.742922 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":56,"thread_start_us":805,"threads_started":1}
I20260925 15:06:55.743391 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.730906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.4.205.195:38253)
0925 15:06:55.735489 (+ 4583us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.735499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.735529 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:06:55.735638 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.735646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.735829 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.736586 (+ 757us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:55.736789 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.737818 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.740626 (+ 2808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.742057 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.742583 (+ 526us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.742791 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.742949 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.743099 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4368,"thread_start_us":4173,"threads_started":1}
I20260925 15:06:55.744361 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.731541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.4.205.194:44655)
0925 15:06:55.732773 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.732782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.733514 (+ 732us) server_negotiation.cc:407] Connection header received
0925 15:06:55.733664 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.733676 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.733850 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.734071 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:55.734708 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.736298 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.740305 (+ 4007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.741722 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.741838 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.742073 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.743904 (+ 1831us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.744076 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1089,"thread_start_us":997,"threads_started":1}
I20260925 15:06:55.745165 5964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.731105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44655 (local address 127.0.0.1:39152)
0925 15:06:55.733269 (+ 2164us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.733284 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.733404 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.734029 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.734036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.734074 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:55.734472 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.734487 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.736440 (+ 1953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.736449 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.740156 (+ 3707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.740168 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.742242 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.742248 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.743695 (+ 1447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.743746 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.744607 (+ 861us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.744742 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2055,"mutex_wait_us":108,"thread_start_us":3316,"threads_started":1}
I20260925 15:06:55.745419 5963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.731452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38115 (local address 127.0.0.1:59732)
0925 15:06:55.734721 (+ 3269us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.734734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.734864 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.735447 (+ 583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.735452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.735476 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:55.735800 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.735812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.737477 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.737482 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.740966 (+ 3484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.740979 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.744397 (+ 3418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.744404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.744575 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.744607 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.744682 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.744777 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3133,"mutex_wait_us":22,"thread_start_us":1590,"threads_started":1}
I20260925 15:06:55.745697 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.732289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59732 (local address 127.4.205.193:38115)
0925 15:06:55.732787 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.732795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.734832 (+ 2037us) server_negotiation.cc:407] Connection header received
0925 15:06:55.735009 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.735019 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.735265 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.735441 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:55.735980 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.737351 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.741169 (+ 3818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.742801 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.742919 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.745061 (+ 2142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.745135 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.745331 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":246,"threads_started":1}
I20260925 15:06:55.747294 5884 tablet_service.cc:1467] Processing CreateTablet for tablet 73a09025b8f146bc8332d1957a514629 (DEFAULT_TABLE table=test-table [id=f8bfd4a0313b481fbe522b7cbe406f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:55.748637 5884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a09025b8f146bc8332d1957a514629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:55.748695 5617 tablet_service.cc:1467] Processing CreateTablet for tablet 73a09025b8f146bc8332d1957a514629 (DEFAULT_TABLE table=test-table [id=f8bfd4a0313b481fbe522b7cbe406f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:55.750458 5617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a09025b8f146bc8332d1957a514629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:55.752135 5751 tablet_service.cc:1467] Processing CreateTablet for tablet 73a09025b8f146bc8332d1957a514629 (DEFAULT_TABLE table=test-table [id=f8bfd4a0313b481fbe522b7cbe406f34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:55.753280 5751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73a09025b8f146bc8332d1957a514629. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:55.759253 5967 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Bootstrap starting.
I20260925 15:06:55.762741 5967 tablet_bootstrap.cc:654] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:55.763989 5967 log.cc:824] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:55.766744 5967 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: No bootstrap required, opened a new log
I20260925 15:06:55.767005 5967 ts_tablet_manager.cc:1397] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:06:55.767460 5970 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Bootstrap starting.
I20260925 15:06:55.769505 5682 heartbeater.cc:503] Master 127.4.205.254:32921 was elected leader, sending a full tablet report...
I20260925 15:06:55.769824 5968 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Bootstrap starting.
I20260925 15:06:55.771270 5970 tablet_bootstrap.cc:654] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:55.772446 5970 log.cc:824] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:55.772972 5968 tablet_bootstrap.cc:654] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:55.774096 5968 log.cc:824] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:55.776633 5970 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: No bootstrap required, opened a new log
I20260925 15:06:55.776852 5968 tablet_bootstrap.cc:492] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: No bootstrap required, opened a new log
I20260925 15:06:55.776866 5970 ts_tablet_manager.cc:1397] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.002s
I20260925 15:06:55.777110 5968 ts_tablet_manager.cc:1397] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.000s
I20260925 15:06:55.778318 5967 raft_consensus.cc:348] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.778678 5967 raft_consensus.cc:374] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:55.778761 5967 raft_consensus.cc:729] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9ec5cf44954f29ba3c1122d6a3de49, State: Initialized, Role: FOLLOWER
I20260925 15:06:55.779295 5967 consensus_queue.cc:260] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.780486 5949 heartbeater.cc:503] Master 127.4.205.254:32921 was elected leader, sending a full tablet report...
I20260925 15:06:55.782539 5967 ts_tablet_manager.cc:1428] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 15:06:55.785632 5970 raft_consensus.cc:348] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.785954 5970 raft_consensus.cc:374] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:55.786059 5970 raft_consensus.cc:729] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9aec845ea5ce49e4afb85e1887d788bb, State: Initialized, Role: FOLLOWER
I20260925 15:06:55.786397 5968 raft_consensus.cc:348] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.786715 5968 raft_consensus.cc:374] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:55.786602 5970 consensus_queue.cc:260] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.786805 5968 raft_consensus.cc:729] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d43d5d5bc56452a89d3662f4107e10f, State: Initialized, Role: FOLLOWER
I20260925 15:06:55.787362 5968 consensus_queue.cc:260] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.787616 5816 heartbeater.cc:503] Master 127.4.205.254:32921 was elected leader, sending a full tablet report...
I20260925 15:06:55.788244 5970 ts_tablet_manager.cc:1428] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Time spent starting tablet: real 0.011s user 0.006s sys 0.005s
I20260925 15:06:55.788931 5968 ts_tablet_manager.cc:1428] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Time spent starting tablet: real 0.012s user 0.008s sys 0.005s
I20260925 15:06:55.823884 5974 raft_consensus.cc:482] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:06:55.824188 5974 raft_consensus.cc:504] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.825695 5974 leader_election.cc:288] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers da9ec5cf44954f29ba3c1122d6a3de49 (127.4.205.195:38253), 8d43d5d5bc56452a89d3662f4107e10f (127.4.205.193:38115)
I20260925 15:06:55.833504 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.826150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44559 (local address 127.4.205.193:38115)
0925 15:06:55.826318 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.826325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.826938 (+ 613us) server_negotiation.cc:407] Connection header received
0925 15:06:55.827382 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.827385 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.827433 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.827494 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:55.828036 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.829013 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.831291 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.832125 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.832705 (+ 580us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.832789 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.833147 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.833267 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:06:55.833576 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.826087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:37339 (local address 127.4.205.195:38253)
0925 15:06:55.826266 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.826273 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.826982 (+ 709us) server_negotiation.cc:407] Connection header received
0925 15:06:55.827110 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.827113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.827169 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.827246 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:55.827820 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.828860 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.831278 (+ 2418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.832046 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.832150 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.832213 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.833172 (+ 959us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.833299 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:06:55.833767 5977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.825931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38115 (local address 127.4.205.194:44559)
0925 15:06:55.826819 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.826843 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.826987 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.827534 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.827544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.827572 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:55.827923 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.827930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.829117 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.829123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.831089 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.831101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.832807 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.832819 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.832972 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.833021 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.833126 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.833247 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":746,"mutex_wait_us":77,"spinlock_wait_cycles":32512,"thread_start_us":550,"threads_started":1}
I20260925 15:06:55.833819 5976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.826059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:38253 (local address 127.4.205.194:37339)
0925 15:06:55.826837 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:55.826851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:55.826987 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:55.827271 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:55.827275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:55.827294 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:55.827624 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.827636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.828966 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.828969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.831096 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:55.831111 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.832829 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.832834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:55.833029 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.833050 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:06:55.833148 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:06:55.833272 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"mutex_wait_us":162,"spinlock_wait_cycles":384,"thread_start_us":400,"threads_started":1}
I20260925 15:06:55.834784 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a09025b8f146bc8332d1957a514629" candidate_uuid: "9aec845ea5ce49e4afb85e1887d788bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" is_pre_election: true
I20260925 15:06:55.834813 5637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a09025b8f146bc8332d1957a514629" candidate_uuid: "9aec845ea5ce49e4afb85e1887d788bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d5d5bc56452a89d3662f4107e10f" is_pre_election: true
I20260925 15:06:55.835341 5637 raft_consensus.cc:2454] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aec845ea5ce49e4afb85e1887d788bb in term 0.
I20260925 15:06:55.835319 5904 raft_consensus.cc:2454] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9aec845ea5ce49e4afb85e1887d788bb in term 0.
I20260925 15:06:55.835963 5704 leader_election.cc:302] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d43d5d5bc56452a89d3662f4107e10f, 9aec845ea5ce49e4afb85e1887d788bb, da9ec5cf44954f29ba3c1122d6a3de49; no voters:
I20260925 15:06:55.836341 5974 raft_consensus.cc:2781] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:06:55.836488 5974 raft_consensus.cc:482] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:06:55.836584 5974 raft_consensus.cc:3037] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:55.838706 5974 raft_consensus.cc:504] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.839246 5974 leader_election.cc:288] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [CANDIDATE]: Term 1 election: Requested vote from peers da9ec5cf44954f29ba3c1122d6a3de49 (127.4.205.195:38253), 8d43d5d5bc56452a89d3662f4107e10f (127.4.205.193:38115)
I20260925 15:06:55.839691 5904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a09025b8f146bc8332d1957a514629" candidate_uuid: "9aec845ea5ce49e4afb85e1887d788bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "da9ec5cf44954f29ba3c1122d6a3de49"
I20260925 15:06:55.839740 5637 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73a09025b8f146bc8332d1957a514629" candidate_uuid: "9aec845ea5ce49e4afb85e1887d788bb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d43d5d5bc56452a89d3662f4107e10f"
I20260925 15:06:55.839887 5904 raft_consensus.cc:3037] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:55.839936 5637 raft_consensus.cc:3037] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:55.842177 5904 raft_consensus.cc:2454] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aec845ea5ce49e4afb85e1887d788bb in term 1.
I20260925 15:06:55.842216 5637 raft_consensus.cc:2454] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9aec845ea5ce49e4afb85e1887d788bb in term 1.
I20260925 15:06:55.842564 5705 leader_election.cc:302] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9aec845ea5ce49e4afb85e1887d788bb, da9ec5cf44954f29ba3c1122d6a3de49; no voters:
I20260925 15:06:55.842875 5974 raft_consensus.cc:2781] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:55.843264 5974 raft_consensus.cc:686] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [term 1 LEADER]: Becoming Leader. State: Replica: 9aec845ea5ce49e4afb85e1887d788bb, State: Running, Role: LEADER
I20260925 15:06:55.843730 5974 consensus_queue.cc:237] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } }
I20260925 15:06:55.848557 5492 catalog_manager.cc:5495] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb reported cstate change: term changed from 0 to 1, leader changed from <none> to 9aec845ea5ce49e4afb85e1887d788bb (127.4.205.194). New cstate: current_term: 1 leader_uuid: "9aec845ea5ce49e4afb85e1887d788bb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "da9ec5cf44954f29ba3c1122d6a3de49" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 38253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9aec845ea5ce49e4afb85e1887d788bb" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44655 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d43d5d5bc56452a89d3662f4107e10f" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38115 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:06:55.872900 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.857033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59746 (local address 127.4.205.193:38115)
0925 15:06:55.857176 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.857180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.857687 (+ 507us) server_negotiation.cc:407] Connection header received
0925 15:06:55.857869 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.857879 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.857937 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.858042 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.859401 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.860118 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.861901 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.862323 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.862441 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.867424 (+ 4983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.867437 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.867451 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.867484 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.870380 (+ 2896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.871013 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.871020 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.871022 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.871139 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.871481 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.871485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.871487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.871812 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.872102 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.872374 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.872574 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
W20260925 15:06:55.911954 5950 tablet.cc:2369] T 73a09025b8f146bc8332d1957a514629 P da9ec5cf44954f29ba3c1122d6a3de49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:06:55.914880 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.902080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.4.205.193:38115)
0925 15:06:55.902310 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.902317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.902528 (+ 211us) server_negotiation.cc:407] Connection header received
0925 15:06:55.902662 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.902665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.902729 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.902808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.903949 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.904639 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.905995 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.906377 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.906472 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.909767 (+ 3295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.909772 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.909775 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.909803 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.912880 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.913390 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.913394 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.913397 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.913530 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.913895 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.913898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.913900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.914128 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.914328 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.914501 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.914643 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
W20260925 15:06:55.945276 5817 tablet.cc:2369] T 73a09025b8f146bc8332d1957a514629 P 9aec845ea5ce49e4afb85e1887d788bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:06:55 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:06:55.955305 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.942611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.4.205.193:38115)
0925 15:06:55.942812 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.942818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.943057 (+ 239us) server_negotiation.cc:407] Connection header received
0925 15:06:55.943276 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.943281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.943337 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.943432 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.944740 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.945692 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.946763 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.947165 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.947253 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.950930 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.950935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.950938 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.950968 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.953119 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.953715 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.953719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.953721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.953829 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.954111 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.954114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.954116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.954375 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.954537 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.954826 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.954973 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 15:06:55.956923 5597 tablet_service.cc:2908] Rejecting scan request for tablet 73a09025b8f146bc8332d1957a514629: Uninitialized: clean time has not yet been initialized
I20260925 15:06:55.959472 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5554
I20260925 15:06:55.971024 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:55.960379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.4.205.193:38115)
0925 15:06:55.960505 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:55.960508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:55.960850 (+ 342us) server_negotiation.cc:407] Connection header received
0925 15:06:55.960961 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:55.960964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:55.961008 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:55.961106 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:55.962193 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.962803 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.963917 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:55.964418 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:55.964517 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:55.966805 (+ 2288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:55.966809 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:55.966812 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:55.966835 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:55.969094 (+ 2259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.969670 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.969673 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.969675 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.969769 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:55.970055 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:55.970058 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:55.970059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:55.970283 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:55.970464 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:55.970671 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:06:55.970807 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:06:55.971580 5677 generic_service.cc:196] Checking for leaks (request via RPC)
W20260925 15:06:56.056058 5683 tablet.cc:2369] T 73a09025b8f146bc8332d1957a514629 P 8d43d5d5bc56452a89d3662f4107e10f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:06:56.056619 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5554
I20260925 15:06:56.072536 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5688
Sep 25 15:06:56 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:06:56.086129 5965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:56.073347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39156 (local address 127.4.205.194:44655)
0925 15:06:56.073522 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:56.073528 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:56.073547 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:06:56.073678 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:56.073689 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:56.073766 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:56.073867 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:56.074998 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:56.075661 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:56.076913 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:56.077325 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:56.077453 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:56.080984 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:56.080995 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:56.081009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:56.081047 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:56.083531 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:56.084152 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:56.084159 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:56.084162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:56.084287 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:56.084644 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:56.084651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:56.084655 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:56.085001 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:56.085271 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:56.085540 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:06:56.085801 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:06:56.086766 5811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:56.174500 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5688
I20260925 15:06:56.191488 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5821
Sep 25 15:06:56 dist-test-slave-64fx krb5kdc[5448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:06:56.206926 5962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:56.192192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47794 (local address 127.4.205.195:38253)
0925 15:06:56.192334 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:56.192338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:56.192405 (+ 67us) server_negotiation.cc:407] Connection header received
0925 15:06:56.192542 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:56.192547 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:56.192620 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:56.192732 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:56.194028 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:56.195146 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:56.196355 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:56.196944 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:56.197057 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:56.200923 (+ 3866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:56.200939 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:56.200958 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:56.200993 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:56.204357 (+ 3364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:56.204933 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:56.204944 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:56.204948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:56.205093 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:56.205399 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:56.205403 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:56.205407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:56.206043 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:56.206314 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:56.206454 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:06:56.206638 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:06:56.207557 5944 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:56.291177 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5821
I20260925 15:06:56.306531 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 5461
I20260925 15:06:56.307570 5523 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:06:56.423527 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 5461
2026-09-25T15:06:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3815 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:06:56 dist-test-slave-64fx krb5kdc[5997](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:06:56 dist-test-slave-64fx krb5kdc[5997](info): set up 2 sockets
Sep 25 15:06:56 dist-test-slave-64fx krb5kdc[5997](info): commencing operation
krb5kdc: starting...
W20260925 15:06:57.473321 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:06:57 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:06:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:06:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:06:57.632576 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44325
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:57.947641 6010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:57.947914 6010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:57.947996 6010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:57.954464 6010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:06:57.954547 6010 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:06:57.954638 6010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:57.954747 6010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:57.954826 6010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:06:57.954936 6010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:06:57.963892 6010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44325
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:57.965040 6010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:57.966292 6010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:57.977222 6016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:57.977280 6018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:57.977247 6015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:57.978209 6010 server_base.cc:1034] running on GCE node
I20260925 15:06:57.978880 6010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:57.980078 6010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:57.981299 6010 hybrid_clock.cc:648] HybridClock initialized: now 1790348817981242 us; error 36 us; skew 500 ppm
I20260925 15:06:57.981649 6010 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:57 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:57.984477 6010 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:06:57.985770 6010 webserver.cc:460] Webserver started at http://127.4.205.254:37019/ using document root <none> and password file <none>
I20260925 15:06:57.986385 6010 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:57.986497 6010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:57.986774 6010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:57.989373 6010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "5878004d445745beb2111d52bfe1dc57"
format_stamp: "Formatted at 2026-09-25 15:06:57 on dist-test-slave-64fx"
server_key: "c19ce786abf6578bfc660d76ba17aa05"
server_key_iv: "848dba9dd77e78da8dee6328fb64abb1"
server_key_version: "encryptionkey@0"
I20260925 15:06:57.990178 6010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "5878004d445745beb2111d52bfe1dc57"
format_stamp: "Formatted at 2026-09-25 15:06:57 on dist-test-slave-64fx"
server_key: "c19ce786abf6578bfc660d76ba17aa05"
server_key_iv: "848dba9dd77e78da8dee6328fb64abb1"
server_key_version: "encryptionkey@0"
I20260925 15:06:57.994819 6010 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:06:57.997952 6025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:57.999116 6010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 15:06:57.999280 6010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "5878004d445745beb2111d52bfe1dc57"
format_stamp: "Formatted at 2026-09-25 15:06:57 on dist-test-slave-64fx"
server_key: "c19ce786abf6578bfc660d76ba17aa05"
server_key_iv: "848dba9dd77e78da8dee6328fb64abb1"
server_key_version: "encryptionkey@0"
I20260925 15:06:57.999468 6010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:58.014915 6010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:58.018754 6010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:58.018982 6010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:58.039080 6010 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44325
I20260925 15:06:58.039099 6086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44325 every 8 connection(s)
I20260925 15:06:58.040318 6010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:06:58.043823 6087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:58.047091 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6010
I20260925 15:06:58.047302 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:06:58.048079 4919 external_mini_cluster.cc:1428] Setting key ebb6cdac81dc7da1d64c275c903d802f
I20260925 15:06:58.056090 6087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Bootstrap starting.
I20260925 15:06:58.059700 6087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:58.060966 6087 log.cc:824] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:06:58 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:58.064162 6087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: No bootstrap required, opened a new log
I20260925 15:06:58.070223 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:58.050015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.4.205.254:44325)
0925 15:06:58.050730 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:58.050740 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:58.050778 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:06:58.052289 (+ 1511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:58.052309 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:58.052687 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:58.053002 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:58.054621 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.055863 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.057494 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.058235 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.058357 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:58.063247 (+ 4890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:58.063254 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:58.063268 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:58.063303 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:58.066341 (+ 3038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.067087 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.067097 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.067102 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.067262 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.067661 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.067665 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.067666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.068146 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:58.068346 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:58.068806 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 15:06:58.069201 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":332,"threads_started":1}
I20260925 15:06:58.073024 6087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } }
I20260925 15:06:58.073378 6087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:58.073472 6087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5878004d445745beb2111d52bfe1dc57, State: Initialized, Role: FOLLOWER
I20260925 15:06:58.074088 6087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } }
I20260925 15:06:58.074317 6087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:06:58.074455 6087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:06:58.074620 6087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:58.076586 6087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } }
I20260925 15:06:58.077088 6087 leader_election.cc:302] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5878004d445745beb2111d52bfe1dc57; no voters:
I20260925 15:06:58.077821 6087 leader_election.cc:288] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:06:58.078118 6092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:58.079049 6092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [term 1 LEADER]: Becoming Leader. State: Replica: 5878004d445745beb2111d52bfe1dc57, State: Running, Role: LEADER
I20260925 15:06:58.079662 6092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } }
I20260925 15:06:58.081620 6093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5878004d445745beb2111d52bfe1dc57. Latest consensus state: current_term: 1 leader_uuid: "5878004d445745beb2111d52bfe1dc57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } } }
I20260925 15:06:58.082021 6093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:58.082113 6087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:06:58.082690 6094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5878004d445745beb2111d52bfe1dc57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5878004d445745beb2111d52bfe1dc57" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44325 } } }
I20260925 15:06:58.083024 6094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57 [sys.catalog]: This master's current role is: LEADER
I20260925 15:06:58.085129 6096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:06:58.090332 6096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:06:58.098898 6096 catalog_manager.cc:1347] Generated new cluster ID: 287935407edc429da34aeba6f2d7bb91
I20260925 15:06:58.099004 6096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:06:58.114070 6096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:06:58.115238 6096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:06:58.120082 6096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Generated new TSK 0
I20260925 15:06:58.120884 6096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:06:58.167042 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44325
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:58.475778 6113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:58.476052 6113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:58.476145 6113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:58.483163 6113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:58.483268 6113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:58.483348 6113 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:06:58.483422 6113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:06:58.493752 6113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:58.494951 6113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:58.496206 6113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:58.508853 6118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:58.508812 6121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:58.508898 6119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:58.509671 6113 server_base.cc:1034] running on GCE node
I20260925 15:06:58.510304 6113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:58.511128 6113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:58.512341 6113 hybrid_clock.cc:648] HybridClock initialized: now 1790348818512265 us; error 54 us; skew 500 ppm
I20260925 15:06:58.512606 6113 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:58 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348818, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:58.515556 6113 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:06:58.516831 6113 webserver.cc:460] Webserver started at http://127.4.205.193:45619/ using document root <none> and password file <none>
I20260925 15:06:58.517469 6113 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:58.517611 6113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:58.517935 6113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:58.520565 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "b4cf715a7d6d479ca0c6e95c90288ee3"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "31e47af6fd22994b83384d0c3af61c92"
server_key_iv: "7f36e9d2ecd8842987f4ed5e5d548fba"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.521347 6113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "b4cf715a7d6d479ca0c6e95c90288ee3"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "31e47af6fd22994b83384d0c3af61c92"
server_key_iv: "7f36e9d2ecd8842987f4ed5e5d548fba"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.525947 6113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:06:58.528757 6128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:58.529872 6113 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:06:58.530025 6113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "b4cf715a7d6d479ca0c6e95c90288ee3"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "31e47af6fd22994b83384d0c3af61c92"
server_key_iv: "7f36e9d2ecd8842987f4ed5e5d548fba"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.530200 6113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:58.540658 6113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:58.544467 6113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:58.544713 6113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:58.546958 6113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:58.547044 6113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:58.547181 6113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:58.547261 6113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:06:58 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348818, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:58.569150 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:58.547354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:44379 (local address 127.4.205.254:44325)
0925 15:06:58.547511 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:58.547516 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:58.548754 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 15:06:58.550967 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:58.550975 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:58.551060 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:58.551189 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:58.553094 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.554225 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.557224 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.557742 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.557874 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:58.562519 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:58.562528 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:58.562535 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:58.562596 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:58.565908 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.566847 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.566854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.566857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.567025 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.567529 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.567537 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.567541 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.567913 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:58.568057 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:58.568663 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 15:06:58.568857 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:06:58.570050 6137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:58.547816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.193:44379)
0925 15:06:58.548598 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:58.548658 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:58.550572 (+ 1914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:58.551461 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:58.551470 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:58.552099 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:58.552927 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:58.552955 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.554409 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.554415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:58.557065 (+ 2650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:58.557083 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.557977 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.557987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:58.558323 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:58.559050 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:58.559070 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:58.562243 (+ 3173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:58.566031 (+ 3788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:58.566041 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:58.566054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:58.566710 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:58.567166 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:58.567176 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:58.567180 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:58.567408 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:58.568055 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:58.568073 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:58.568345 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 15:06:58.568564 (+ 219us) client_negotiation.cc:240] Negotiation successful
0925 15:06:58.568963 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":351,"threads_started":1}
I20260925 15:06:58.588583 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:58.575413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:41821 (local address 127.4.205.254:44325)
0925 15:06:58.575568 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:58.575574 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:58.576026 (+ 452us) server_negotiation.cc:407] Connection header received
0925 15:06:58.576203 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:58.576208 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:58.576269 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:58.576350 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:58.577177 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.577894 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.580335 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.580859 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.580990 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:58.583360 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:58.583364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:58.583366 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:58.583389 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:58.585954 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.586789 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.586795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.586800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.586935 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:58.587320 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:58.587327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:58.587331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:58.587567 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:58.587712 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:58.588079 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:06:58.588262 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:06:58.588697 6204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:58.575432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.193:41821)
0925 15:06:58.575937 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:58.575956 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:58.576073 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:58.576368 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:58.576372 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:58.576790 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:58.577049 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:58.577056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.578060 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.578065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:58.580167 (+ 2102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:58.580187 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:58.581056 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:58.581072 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:58.581331 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:58.582019 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:58.582047 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:58.583200 (+ 1153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:58.586068 (+ 2868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:58.586073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:58.586078 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:58.586669 (+ 591us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:58.587068 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:58.587073 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:58.587082 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:58.587224 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:58.587707 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:58.587711 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:58.587872 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:06:58.588007 (+ 135us) client_negotiation.cc:240] Negotiation successful
0925 15:06:58.588281 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":313,"threads_started":1}
I20260925 15:06:58.597457 6113 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:40085
I20260925 15:06:58.597488 6247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:40085 every 8 connection(s)
I20260925 15:06:58.598515 6113 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:06:58.600219 6248 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44325
I20260925 15:06:58.600466 6248 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:58.601019 6248 heartbeater.cc:511] Master 127.4.205.254:44325 requested a full tablet report, sending...
I20260925 15:06:58.602458 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6113
I20260925 15:06:58.602685 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:06:58.602864 6042 ts_manager.cc:194] Registered new tserver with Master: b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193:40085)
I20260925 15:06:58.603379 4919 external_mini_cluster.cc:1428] Setting key 1bce50dcd708b361a912672610dc36b8
I20260925 15:06:58.604517 6042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:44379
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:06:58.638798 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44325
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:58.943475 6252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:58.943756 6252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:58.943847 6252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:58.950881 6252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:58.950989 6252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:58.951071 6252 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:06:58.951155 6252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:06:58.960686 6252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:58.962067 6252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:58.963322 6252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:58.976078 6257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:58.976100 6258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:58.976047 6260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:58.976929 6252 server_base.cc:1034] running on GCE node
I20260925 15:06:58.977597 6252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:58.978488 6252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:58.979722 6252 hybrid_clock.cc:648] HybridClock initialized: now 1790348818979648 us; error 52 us; skew 500 ppm
I20260925 15:06:58.980014 6252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:58 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348818, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:58.983088 6252 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:06:58.984447 6252 webserver.cc:460] Webserver started at http://127.4.205.194:46387/ using document root <none> and password file <none>
I20260925 15:06:58.985127 6252 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:58.985247 6252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:58.985549 6252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:58.988541 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "6f3f5e9765364593819cf579babd6928"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "aaf7d153091f9b7e4e125dfa2185a903"
server_key_iv: "dc3b84f124f57c80a4fbddedc940a66d"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.989318 6252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "6f3f5e9765364593819cf579babd6928"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "aaf7d153091f9b7e4e125dfa2185a903"
server_key_iv: "dc3b84f124f57c80a4fbddedc940a66d"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.994233 6252 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 15:06:58.997761 6267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:58.999060 6252 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:06:58.999230 6252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "6f3f5e9765364593819cf579babd6928"
format_stamp: "Formatted at 2026-09-25 15:06:58 on dist-test-slave-64fx"
server_key: "aaf7d153091f9b7e4e125dfa2185a903"
server_key_iv: "dc3b84f124f57c80a4fbddedc940a66d"
server_key_version: "encryptionkey@0"
I20260925 15:06:58.999459 6252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:59.017309 6252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:59.021895 6252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:59.022161 6252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:59.024686 6252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:59.024792 6252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:59.024952 6252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:59.025027 6252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:06:59 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348818, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:59.054030 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.025412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:52353 (local address 127.4.205.254:44325)
0925 15:06:59.025635 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.025641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.027121 (+ 1480us) server_negotiation.cc:407] Connection header received
0925 15:06:59.029448 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.029453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.029518 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.029641 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:59.032122 (+ 2481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.033469 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.035826 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.036634 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.036817 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.048259 (+ 11442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:59.048264 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:59.048267 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:59.048297 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:59.051099 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.051795 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.051802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.051806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.051943 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.052368 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.052374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.052379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.052647 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:59.052783 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.053442 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.053707 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:06:59.055235 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.026027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.194:52353)
0925 15:06:59.026927 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.026980 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.029074 (+ 2094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.030183 (+ 1109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.030197 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.031059 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:59.031890 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.031934 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.033687 (+ 1753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.033695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.035646 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.035665 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.037355 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.037367 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.037790 (+ 423us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.045760 (+ 7970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:59.045782 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:59.048022 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:59.051230 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.051239 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.051256 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.051677 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.052049 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.052054 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.052058 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.052263 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.052776 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:59.052783 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:59.053106 (+ 323us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.053356 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.053975 (+ 619us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":377,"threads_started":1}
I20260925 15:06:59.080322 6252 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:44209
I20260925 15:06:59.080402 6386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:44209 every 8 connection(s)
I20260925 15:06:59.081440 6349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.061105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.194:48755)
0925 15:06:59.061734 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.061749 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.061880 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.062181 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.062185 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.062631 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:59.062910 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.062918 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.064060 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.064070 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.070153 (+ 6083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.070178 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.071553 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.071558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.071836 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.072461 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:59.072488 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:59.073858 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:59.078157 (+ 4299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.078165 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.078174 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.078704 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.079111 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.079118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.079124 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.079312 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.080033 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:59.080039 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:59.080288 (+ 249us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.080497 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.080877 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":317,"threads_started":1}
I20260925 15:06:59.081985 6252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:06:59.082589 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.060905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:48755 (local address 127.4.205.254:44325)
0925 15:06:59.061045 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.061049 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.061820 (+ 771us) server_negotiation.cc:407] Connection header received
0925 15:06:59.061985 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.061991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.062049 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.062161 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:59.063044 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.063909 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.070349 (+ 6440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.071122 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.071278 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.074124 (+ 2846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:59.074132 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:59.074140 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:59.074182 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:59.077939 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.078833 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.078839 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.078842 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.078974 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.079445 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.079453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.079459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.079906 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:59.080052 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.081902 (+ 1850us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.082125 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:06:59.084373 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6252
I20260925 15:06:59.084582 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:06:59.085047 4919 external_mini_cluster.cc:1428] Setting key 80ddfb792335b154643877d00baf8329
I20260925 15:06:59.085052 6387 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44325
I20260925 15:06:59.087514 6387 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:59.088567 6387 heartbeater.cc:511] Master 127.4.205.254:44325 requested a full tablet report, sending...
I20260925 15:06:59.090090 6042 ts_manager.cc:194] Registered new tserver with Master: 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209)
I20260925 15:06:59.091292 6042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:52353
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:06:59.122802 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44325
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:06:59.430224 6391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:06:59.430505 6391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:06:59.430600 6391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:06:59.441102 6391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:06:59.441210 6391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:06:59.441293 6391 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:06:59.441373 6391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:06:59.451280 6391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:44325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:06:59.452575 6391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:06:59.453948 6391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:06:59.466800 6396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:59.466841 6397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:06:59.466876 6399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:06:59.467657 6391 server_base.cc:1034] running on GCE node
I20260925 15:06:59.468276 6391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:06:59.469173 6391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:06:59.470425 6391 hybrid_clock.cc:648] HybridClock initialized: now 1790348819470339 us; error 51 us; skew 500 ppm
I20260925 15:06:59.470722 6391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:06:59 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:06:59.473755 6391 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:06:59.475033 6391 webserver.cc:460] Webserver started at http://127.4.205.195:46397/ using document root <none> and password file <none>
I20260925 15:06:59.475672 6391 fs_manager.cc:359] Metadata directory not provided
I20260925 15:06:59.475752 6391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:06:59.475991 6391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:06:59.478812 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "82061903c4ff459298a3c0e2a0d3da42"
format_stamp: "Formatted at 2026-09-25 15:06:59 on dist-test-slave-64fx"
server_key: "bf1601f8196dc80a5f5b7c8ae91026c9"
server_key_iv: "1d9035791be3ca9efd58add808dcfa39"
server_key_version: "encryptionkey@0"
I20260925 15:06:59.479496 6391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "82061903c4ff459298a3c0e2a0d3da42"
format_stamp: "Formatted at 2026-09-25 15:06:59 on dist-test-slave-64fx"
server_key: "bf1601f8196dc80a5f5b7c8ae91026c9"
server_key_iv: "1d9035791be3ca9efd58add808dcfa39"
server_key_version: "encryptionkey@0"
I20260925 15:06:59.484270 6391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:06:59.487622 6406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:59.488747 6391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:06:59.488902 6391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "82061903c4ff459298a3c0e2a0d3da42"
format_stamp: "Formatted at 2026-09-25 15:06:59 on dist-test-slave-64fx"
server_key: "bf1601f8196dc80a5f5b7c8ae91026c9"
server_key_iv: "1d9035791be3ca9efd58add808dcfa39"
server_key_version: "encryptionkey@0"
I20260925 15:06:59.489092 6391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:06:59.511780 6391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:06:59.515682 6391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:06:59.515940 6391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:06:59.518507 6391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:06:59.518620 6391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:06:59.518759 6391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:06:59.518831 6391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:06:59 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:59.541229 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.519171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:33771 (local address 127.4.205.254:44325)
0925 15:06:59.519354 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.519360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.520803 (+ 1443us) server_negotiation.cc:407] Connection header received
0925 15:06:59.522844 (+ 2041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.522849 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.522937 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.523044 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:59.525760 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.526705 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.529209 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.529806 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.529932 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.534717 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:59.534727 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:59.534733 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:59.534778 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:59.537822 (+ 3044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.538596 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.538605 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.538611 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.538816 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.539240 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.539247 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.539252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.539666 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:59.539845 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.540411 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.540702 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:06:59.541667 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.519730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.195:33771)
0925 15:06:59.520609 (+ 879us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.520662 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.522548 (+ 1886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.523440 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.523456 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.524407 (+ 951us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:59.525480 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.525516 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.526929 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.526937 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.529067 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.529077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.530029 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.530037 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.530275 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.530992 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:59.531016 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:59.534476 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:59.537975 (+ 3499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.537981 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.537991 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.538461 (+ 470us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.538929 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.538932 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.538934 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.539112 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.539784 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:59.539788 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:59.540063 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.540274 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.540578 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":394,"threads_started":1}
I20260925 15:06:59.562371 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.548591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:37003 (local address 127.4.205.254:44325)
0925 15:06:59.548803 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.548811 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.550231 (+ 1420us) server_negotiation.cc:407] Connection header received
0925 15:06:59.550336 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.550345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.550434 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.550554 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:59.551328 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.552669 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.554169 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.554957 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.555101 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.556682 (+ 1581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:59.556691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:59.556698 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:59.556743 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:59.559563 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.560459 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.560463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.560465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.560569 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.560972 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.560975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.560977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.561261 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:59.561361 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.561954 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.562095 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:06:59.562865 6489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.548296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.4.205.195:37003)
0925 15:06:59.550130 (+ 1834us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.550153 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.550249 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.550558 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.550562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.550940 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:06:59.551208 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.551216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.552799 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.552803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.553923 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.553933 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.555132 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.555136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.555265 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.555850 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:06:59.555865 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:06:59.556513 (+ 648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:06:59.559690 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.559698 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.559705 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.560343 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.560671 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:06:59.560678 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:06:59.560684 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:06:59.560871 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:06:59.561370 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:06:59.561377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:06:59.561645 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.561844 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.562263 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1682,"thread_start_us":1539,"threads_started":1}
I20260925 15:06:59.574147 6391 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:46847
I20260925 15:06:59.574168 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:46847 every 8 connection(s)
I20260925 15:06:59.575351 6391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:06:59.577160 6526 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44325
I20260925 15:06:59.577363 6526 heartbeater.cc:464] Registering TS with master...
I20260925 15:06:59.577821 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6391
I20260925 15:06:59.577992 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:06:59.578047 6526 heartbeater.cc:511] Master 127.4.205.254:44325 requested a full tablet report, sending...
I20260925 15:06:59.578416 4919 external_mini_cluster.cc:1428] Setting key 953c2bd23347e220757156a0c33a0ce3
I20260925 15:06:59.579370 6042 ts_manager.cc:194] Registered new tserver with Master: 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847)
I20260925 15:06:59.580358 6042 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:33771
I20260925 15:06:59.581090 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:06:59 dist-test-slave-64fx krb5kdc[5997](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 15:06:59.606941 6248 heartbeater.cc:503] Master 127.4.205.254:44325 was elected leader, sending a full tablet report...
Sep 25 15:06:59 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:06:59.625972 6090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.612791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48988 (local address 127.4.205.254:44325)
0925 15:06:59.613013 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.613022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.613314 (+ 292us) server_negotiation.cc:407] Connection header received
0925 15:06:59.613512 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.613516 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.613633 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.613730 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:06:59.614991 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.615711 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.616955 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.617352 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.617474 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.621036 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:06:59.621040 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:06:59.621043 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:06:59.621074 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:06:59.623552 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.624157 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.624163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.624172 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.624277 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:06:59.624617 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:06:59.624624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:06:59.624629 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:06:59.624874 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:06:59.625116 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.625442 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.625670 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:06:59.629405 6041 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:06:59.636071 6041 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:06:59.661652 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.647564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44209 (local address 127.0.0.1:57174)
0925 15:06:59.653738 (+ 6174us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.653762 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.653946 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.654684 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.654692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.654716 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:59.655155 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.655169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.656804 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.656811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.658734 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.658747 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.660634 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.660641 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.660861 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.660921 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.661046 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.661265 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6026,"thread_start_us":5846,"threads_started":1}
I20260925 15:06:59.662467 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.648438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57174 (local address 127.4.205.194:44209)
0925 15:06:59.648935 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.648943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.653852 (+ 4909us) server_negotiation.cc:407] Connection header received
0925 15:06:59.654102 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.654113 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.654388 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.654575 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:59.655350 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.656690 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.658897 (+ 2207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.660149 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.660303 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.660514 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.662026 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.662167 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":250,"threads_started":1}
I20260925 15:06:59.662503 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.648028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.4.205.195:46847)
0925 15:06:59.654169 (+ 6141us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.654179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.654211 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:06:59.654310 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.654317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.654595 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.654897 (+ 302us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:59.655559 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.657073 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.659477 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.660929 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.661754 (+ 825us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.661958 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.662032 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.662190 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5941,"thread_start_us":5733,"threads_started":1}
I20260925 15:06:59.661980 6539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.647333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:46847 (local address 127.0.0.1:49672)
0925 15:06:59.648112 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.648147 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.648344 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.654881 (+ 6537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.654885 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.654910 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:59.655349 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.655362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.657214 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.657221 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.659318 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.659333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.661195 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.661202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.661360 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.661396 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.661458 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.661618 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":1654,"threads_started":1}
I20260925 15:06:59.663121 6541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.647947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40085 (local address 127.0.0.1:43574)
0925 15:06:59.654404 (+ 6457us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.654422 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.654530 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.656257 (+ 1727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.656262 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.656299 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:59.656620 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.656630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.658555 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.658560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.660449 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.660464 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.662325 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.662332 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.662497 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.662531 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.662644 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.662787 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6339,"thread_start_us":6213,"threads_started":1}
I20260925 15:06:59.663367 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.648511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43574 (local address 127.4.205.193:40085)
0925 15:06:59.655737 (+ 7226us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.655746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.655771 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:06:59.655865 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.655873 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.656069 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.656792 (+ 723us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:59.656947 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.658434 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.660625 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.662108 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.662242 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.662441 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.662817 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.662966 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7047,"thread_start_us":6925,"threads_started":1}
I20260925 15:06:59.666347 6456 tablet_service.cc:1467] Processing CreateTablet for tablet d06349de7e9e438f83dee025a0ff9516 (DEFAULT_TABLE table=test-table [id=bada37cffea443ff9939d9f0b4d262fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:59.666352 6178 tablet_service.cc:1467] Processing CreateTablet for tablet d06349de7e9e438f83dee025a0ff9516 (DEFAULT_TABLE table=test-table [id=bada37cffea443ff9939d9f0b4d262fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:59.666352 6317 tablet_service.cc:1467] Processing CreateTablet for tablet d06349de7e9e438f83dee025a0ff9516 (DEFAULT_TABLE table=test-table [id=bada37cffea443ff9939d9f0b4d262fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:06:59.667470 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06349de7e9e438f83dee025a0ff9516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:59.667495 6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06349de7e9e438f83dee025a0ff9516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:59.667505 6317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d06349de7e9e438f83dee025a0ff9516. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:06:59.678192 6545 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Bootstrap starting.
I20260925 15:06:59.681507 6547 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Bootstrap starting.
I20260925 15:06:59.681732 6545 tablet_bootstrap.cc:654] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:59.682224 6546 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Bootstrap starting.
I20260925 15:06:59.683440 6545 log.cc:824] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:59.685889 6545 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: No bootstrap required, opened a new log
I20260925 15:06:59.685997 6546 tablet_bootstrap.cc:654] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:59.686167 6545 ts_tablet_manager.cc:1397] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260925 15:06:59.687186 6546 log.cc:824] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:59.687759 6547 tablet_bootstrap.cc:654] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Neither blocks nor log segments found. Creating new log.
I20260925 15:06:59.689182 6547 log.cc:824] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Log is configured to *not* fsync() on all Append() calls
I20260925 15:06:59.692355 6546 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: No bootstrap required, opened a new log
I20260925 15:06:59.692654 6546 ts_tablet_manager.cc:1397] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260925 15:06:59.694446 6545 raft_consensus.cc:348] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.694765 6545 raft_consensus.cc:374] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:59.694872 6545 raft_consensus.cc:729] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82061903c4ff459298a3c0e2a0d3da42, State: Initialized, Role: FOLLOWER
I20260925 15:06:59.695441 6545 consensus_queue.cc:260] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.696390 6526 heartbeater.cc:503] Master 127.4.205.254:44325 was elected leader, sending a full tablet report...
I20260925 15:06:59.696970 6545 ts_tablet_manager.cc:1428] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Time spent starting tablet: real 0.011s user 0.009s sys 0.001s
I20260925 15:06:59.697955 6547 tablet_bootstrap.cc:492] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: No bootstrap required, opened a new log
I20260925 15:06:59.698529 6547 ts_tablet_manager.cc:1397] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.006s
I20260925 15:06:59.703644 6546 raft_consensus.cc:348] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.704054 6546 raft_consensus.cc:374] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:59.704159 6546 raft_consensus.cc:729] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cf715a7d6d479ca0c6e95c90288ee3, State: Initialized, Role: FOLLOWER
I20260925 15:06:59.705101 6546 consensus_queue.cc:260] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.706630 6546 ts_tablet_manager.cc:1428] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260925 15:06:59.708227 6547 raft_consensus.cc:348] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.708536 6547 raft_consensus.cc:374] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:06:59.708657 6547 raft_consensus.cc:729] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3f5e9765364593819cf579babd6928, State: Initialized, Role: FOLLOWER
I20260925 15:06:59.709110 6547 consensus_queue.cc:260] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.709997 6387 heartbeater.cc:503] Master 127.4.205.254:44325 was elected leader, sending a full tablet report...
I20260925 15:06:59.710441 6547 ts_tablet_manager.cc:1428] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
W20260925 15:06:59.826843 6527 tablet.cc:2369] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:06:59.834982 6388 tablet.cc:2369] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:06:59.850723 6249 tablet.cc:2369] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:06:59.867743 6552 raft_consensus.cc:482] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:06:59.868032 6552 raft_consensus.cc:504] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.869663 6552 leader_election.cc:288] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847), 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209)
I20260925 15:06:59.876768 6542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.869974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42015 (local address 127.4.205.195:46847)
0925 15:06:59.870166 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.870174 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.871490 (+ 1316us) server_negotiation.cc:407] Connection header received
0925 15:06:59.871709 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.871715 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.871779 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.871863 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:59.872417 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.873172 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.875035 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.875863 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.876040 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.876114 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.876388 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.876520 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:06:59.876760 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.870245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37107 (local address 127.4.205.194:44209)
0925 15:06:59.870490 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.870498 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.871571 (+ 1073us) server_negotiation.cc:407] Connection header received
0925 15:06:59.871807 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.871813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.871874 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.871968 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:06:59.872569 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.873325 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.875051 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.875829 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.876013 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.876086 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.876362 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.876500 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 15:06:59.876929 6554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.870015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:46847 (local address 127.4.205.193:42015)
0925 15:06:59.871393 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.871411 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.871564 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.871877 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.871881 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.871902 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:59.872281 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.872293 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.873285 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.873293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.874895 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.874906 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.876077 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.876083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.876246 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.876268 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.876365 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.876516 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1231,"mutex_wait_us":140,"thread_start_us":933,"threads_started":1}
I20260925 15:06:59.877298 6555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.870102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44209 (local address 127.4.205.193:37107)
0925 15:06:59.871477 (+ 1375us) negotiation.cc:107] Waiting for socket to connect
0925 15:06:59.871491 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:06:59.871666 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:06:59.871980 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:06:59.871984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:06:59.872019 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:06:59.872434 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.872441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.873456 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.873464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.874912 (+ 1448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:06:59.874923 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.876037 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.876043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:06:59.876202 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.876231 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:06:59.876339 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:06:59.876499 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"mutex_wait_us":59,"thread_start_us":922,"threads_started":1}
I20260925 15:06:59.877861 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06349de7e9e438f83dee025a0ff9516" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42" is_pre_election: true
I20260925 15:06:59.878208 6337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06349de7e9e438f83dee025a0ff9516" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3f5e9765364593819cf579babd6928" is_pre_election: true
I20260925 15:06:59.878415 6480 raft_consensus.cc:2454] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 0.
I20260925 15:06:59.878628 6337 raft_consensus.cc:2454] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 0.
I20260925 15:06:59.879050 6129 leader_election.cc:302] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82061903c4ff459298a3c0e2a0d3da42, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:06:59.879386 6552 raft_consensus.cc:2781] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:06:59.879522 6552 raft_consensus.cc:482] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:06:59.879616 6552 raft_consensus.cc:3037] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:59.882383 6552 raft_consensus.cc:504] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.882892 6552 leader_election.cc:288] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 election: Requested vote from peers 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847), 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209)
I20260925 15:06:59.883404 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06349de7e9e438f83dee025a0ff9516" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42"
I20260925 15:06:59.883457 6337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d06349de7e9e438f83dee025a0ff9516" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3f5e9765364593819cf579babd6928"
I20260925 15:06:59.883572 6480 raft_consensus.cc:3037] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:59.883678 6337 raft_consensus.cc:3037] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:06:59.885679 6480 raft_consensus.cc:2454] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 1.
I20260925 15:06:59.885756 6337 raft_consensus.cc:2454] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 1.
I20260925 15:06:59.886045 6129 leader_election.cc:302] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82061903c4ff459298a3c0e2a0d3da42, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:06:59.886314 6552 raft_consensus.cc:2781] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:06:59.886718 6552 raft_consensus.cc:686] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 LEADER]: Becoming Leader. State: Replica: b4cf715a7d6d479ca0c6e95c90288ee3, State: Running, Role: LEADER
I20260925 15:06:59.887135 6552 consensus_queue.cc:237] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } }
I20260925 15:06:59.891247 6041 catalog_manager.cc:5495] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } health_report { overall_health: HEALTHY } } }
I20260925 15:06:59.955492 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:06:59.950454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43578 (local address 127.4.205.193:40085)
0925 15:06:59.950619 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:06:59.950624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:06:59.950639 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:06:59.950873 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:06:59.950876 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:06:59.950954 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:06:59.951071 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:06:59.951348 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.951991 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.953179 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:06:59.954090 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:06:59.954184 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:06:59.954572 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:06:59.955037 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:06:59.955138 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:06:59.955242 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:06:59.955306 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:06:59.962875 6480 raft_consensus.cc:1261] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Refusing update from remote peer b4cf715a7d6d479ca0c6e95c90288ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:06:59.962898 6337 raft_consensus.cc:1261] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Refusing update from remote peer b4cf715a7d6d479ca0c6e95c90288ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:06:59.963795 6552 consensus_queue.cc:1035] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:06:59.964072 6556 consensus_queue.cc:1035] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:06:59.973973 6561 mvcc.cc:205] Tried to move back new op lower bound from 7333268766559113216 to 7333268766260854784. Current Snapshot: MvccSnapshot[applied={T|T < 7333268766559113216}]
I20260925 15:06:59.974629 6559 mvcc.cc:205] Tried to move back new op lower bound from 7333268766559113216 to 7333268766260854784. Current Snapshot: MvccSnapshot[applied={T|T < 7333268766559113216}]
I20260925 15:06:59.994210 6042 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:06:59.994477 6042 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:06:59.997928 6042 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Sending DeleteTablet for 3 replicas of tablet d06349de7e9e438f83dee025a0ff9516
I20260925 15:06:59.998993 6456 tablet_service.cc:1514] Processing DeleteTablet for tablet d06349de7e9e438f83dee025a0ff9516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49672
I20260925 15:06:59.998994 6317 tablet_service.cc:1514] Processing DeleteTablet for tablet d06349de7e9e438f83dee025a0ff9516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:57174
I20260925 15:06:59.999050 6178 tablet_service.cc:1514] Processing DeleteTablet for tablet d06349de7e9e438f83dee025a0ff9516 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:06:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:43574
I20260925 15:07:00.006994 6042 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:00.008047 6042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:00.012547 6569 tablet_replica.cc:330] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: stopping tablet replica
I20260925 15:07:00.013144 6569 raft_consensus.cc:2229] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:00.013729 6569 raft_consensus.cc:2258] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.014515 6568 tablet_replica.cc:330] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: stopping tablet replica
I20260925 15:07:00.015023 6568 raft_consensus.cc:2229] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:00.016681 6177 tablet_service.cc:1467] Processing CreateTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb (DEFAULT_TABLE table=test-table [id=aeafa150731a4d7e827e303c2682cf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:00.017041 6456 tablet_service.cc:1467] Processing CreateTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb (DEFAULT_TABLE table=test-table [id=aeafa150731a4d7e827e303c2682cf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:00.017275 6570 tablet_replica.cc:330] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: stopping tablet replica
I20260925 15:07:00.017324 6177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8550aefc92e4095bacacf0d6fa0cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.017445 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8550aefc92e4095bacacf0d6fa0cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.017683 6570 raft_consensus.cc:2229] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:00.018304 6570 raft_consensus.cc:2258] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.019644 6569 ts_tablet_manager.cc:1905] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.016553 6316 tablet_service.cc:1467] Processing CreateTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb (DEFAULT_TABLE table=test-table [id=aeafa150731a4d7e827e303c2682cf9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:00.024940 6545 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Bootstrap starting.
I20260925 15:07:00.025229 6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8550aefc92e4095bacacf0d6fa0cfcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.025200 6568 raft_consensus.cc:2258] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.025952 6546 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Bootstrap starting.
I20260925 15:07:00.028964 6546 tablet_bootstrap.cc:654] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.029217 6545 tablet_bootstrap.cc:654] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.029378 6568 ts_tablet_manager.cc:1905] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.030637 6570 ts_tablet_manager.cc:1905] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.031867 6547 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Bootstrap starting.
I20260925 15:07:00.033752 6547 tablet_bootstrap.cc:654] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.038309 6569 ts_tablet_manager.cc:1918] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:00.042232 6568 ts_tablet_manager.cc:1918] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:00.042390 6568 log.cc:1196] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/d06349de7e9e438f83dee025a0ff9516
I20260925 15:07:00.042860 6568 ts_tablet_manager.cc:1939] T d06349de7e9e438f83dee025a0ff9516 P 6f3f5e9765364593819cf579babd6928: Deleting consensus metadata
I20260925 15:07:00.038475 6545 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: No bootstrap required, opened a new log
I20260925 15:07:00.044368 6569 log.cc:1196] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/d06349de7e9e438f83dee025a0ff9516
I20260925 15:07:00.044643 6027 catalog_manager.cc:4841] TS 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209): tablet d06349de7e9e438f83dee025a0ff9516 (table test-table [id=bada37cffea443ff9939d9f0b4d262fd]) successfully deleted
I20260925 15:07:00.044656 6545 ts_tablet_manager.cc:1397] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.006s
I20260925 15:07:00.045738 6545 raft_consensus.cc:348] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.046092 6545 raft_consensus.cc:374] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.046249 6545 raft_consensus.cc:729] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82061903c4ff459298a3c0e2a0d3da42, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.046597 6545 consensus_queue.cc:260] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.047390 6545 ts_tablet_manager.cc:1428] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:00.047992 6569 ts_tablet_manager.cc:1939] T d06349de7e9e438f83dee025a0ff9516 P 82061903c4ff459298a3c0e2a0d3da42: Deleting consensus metadata
I20260925 15:07:00.049636 6026 catalog_manager.cc:4841] TS 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847): tablet d06349de7e9e438f83dee025a0ff9516 (table test-table [id=bada37cffea443ff9939d9f0b4d262fd]) successfully deleted
I20260925 15:07:00.058589 6547 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: No bootstrap required, opened a new log
I20260925 15:07:00.058763 6547 ts_tablet_manager.cc:1397] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.014s
I20260925 15:07:00.059911 6570 ts_tablet_manager.cc:1918] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:00.059801 6547 raft_consensus.cc:348] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.060075 6570 log.cc:1196] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/d06349de7e9e438f83dee025a0ff9516
I20260925 15:07:00.060091 6547 raft_consensus.cc:374] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.060196 6547 raft_consensus.cc:729] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3f5e9765364593819cf579babd6928, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.060645 6570 ts_tablet_manager.cc:1939] T d06349de7e9e438f83dee025a0ff9516 P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting consensus metadata
I20260925 15:07:00.060552 6547 consensus_queue.cc:260] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.062333 6029 catalog_manager.cc:4841] TS b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193:40085): tablet d06349de7e9e438f83dee025a0ff9516 (table test-table [id=bada37cffea443ff9939d9f0b4d262fd]) successfully deleted
I20260925 15:07:00.062929 6547 ts_tablet_manager.cc:1428] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:07:00.064355 6546 tablet_bootstrap.cc:492] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: No bootstrap required, opened a new log
I20260925 15:07:00.064507 6546 ts_tablet_manager.cc:1397] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent bootstrapping tablet: real 0.039s user 0.018s sys 0.009s
I20260925 15:07:00.065387 6546 raft_consensus.cc:348] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.065621 6546 raft_consensus.cc:374] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.065711 6546 raft_consensus.cc:729] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cf715a7d6d479ca0c6e95c90288ee3, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.065958 6546 consensus_queue.cc:260] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.066561 6546 ts_tablet_manager.cc:1428] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:00.399428 6553 raft_consensus.cc:482] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:00.400205 6553 raft_consensus.cc:504] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.401607 6553 leader_election.cc:288] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193:40085), 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847)
I20260925 15:07:00.411222 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.402215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:57793 (local address 127.4.205.193:40085)
0925 15:07:00.402473 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.402480 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.402500 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:07:00.402570 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.402577 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.402677 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.402756 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:00.406285 (+ 3529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.407232 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.409199 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.410169 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.410326 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.410417 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.410776 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.410910 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260925 15:07:00.411238 6574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.401808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40085 (local address 127.4.205.194:57793)
0925 15:07:00.402260 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.402279 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.402432 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.405664 (+ 3232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.405672 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.405708 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:00.406094 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.406109 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.407352 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.407359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.409049 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.409060 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.410360 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.410366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.410565 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.410588 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.410731 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.410862 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":452,"threads_started":1}
I20260925 15:07:00.412359 6202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8550aefc92e4095bacacf0d6fa0cfcb" candidate_uuid: "6f3f5e9765364593819cf579babd6928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" is_pre_election: true
I20260925 15:07:00.412562 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.401831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:46847 (local address 127.4.205.194:42599)
0925 15:07:00.402861 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.402877 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.402993 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.406808 (+ 3815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.406812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.406833 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:00.407120 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.407127 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.408526 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.408532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.410529 (+ 1997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.410543 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.411845 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.411851 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.411987 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.412031 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.412141 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.412261 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":926,"thread_start_us":1018,"threads_started":1}
I20260925 15:07:00.412843 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.402174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:42599 (local address 127.4.205.195:46847)
0925 15:07:00.406571 (+ 4397us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.406575 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.406590 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:00.406641 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.406644 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.406692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.406805 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:00.407277 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.408386 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.410699 (+ 2313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.411668 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.411828 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.411889 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.412187 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.412316 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4304,"thread_start_us":4201,"threads_started":1}
I20260925 15:07:00.412726 6202 raft_consensus.cc:2454] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3f5e9765364593819cf579babd6928 in term 0.
I20260925 15:07:00.413293 6271 leader_election.cc:302] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3f5e9765364593819cf579babd6928, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:07:00.413352 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8550aefc92e4095bacacf0d6fa0cfcb" candidate_uuid: "6f3f5e9765364593819cf579babd6928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42" is_pre_election: true
I20260925 15:07:00.413583 6480 raft_consensus.cc:2454] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f3f5e9765364593819cf579babd6928 in term 0.
I20260925 15:07:00.413745 6553 raft_consensus.cc:2781] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:00.413889 6553 raft_consensus.cc:482] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:00.413995 6553 raft_consensus.cc:3037] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.416440 6553 raft_consensus.cc:504] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.417037 6553 leader_election.cc:288] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [CANDIDATE]: Term 1 election: Requested vote from peers b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193:40085), 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847)
I20260925 15:07:00.417495 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8550aefc92e4095bacacf0d6fa0cfcb" candidate_uuid: "6f3f5e9765364593819cf579babd6928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42"
I20260925 15:07:00.417495 6202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8550aefc92e4095bacacf0d6fa0cfcb" candidate_uuid: "6f3f5e9765364593819cf579babd6928" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3"
I20260925 15:07:00.417725 6202 raft_consensus.cc:3037] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.417722 6480 raft_consensus.cc:3037] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.420506 6480 raft_consensus.cc:2454] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3f5e9765364593819cf579babd6928 in term 1.
I20260925 15:07:00.420506 6202 raft_consensus.cc:2454] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f3f5e9765364593819cf579babd6928 in term 1.
I20260925 15:07:00.420873 6271 leader_election.cc:302] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f3f5e9765364593819cf579babd6928, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:07:00.421175 6553 raft_consensus.cc:2781] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:00.421592 6553 raft_consensus.cc:686] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 1 LEADER]: Becoming Leader. State: Replica: 6f3f5e9765364593819cf579babd6928, State: Running, Role: LEADER
I20260925 15:07:00.421957 6553 consensus_queue.cc:237] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.424969 6042 catalog_manager.cc:5495] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f3f5e9765364593819cf579babd6928 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "6f3f5e9765364593819cf579babd6928" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } health_report { overall_health: UNKNOWN } } }
Sep 25 15:07:00 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348817, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:00.532419 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.519612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44325 (local address 127.0.0.1:48996)
0925 15:07:00.520246 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.520263 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.520374 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.520659 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.520662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.521364 (+ 702us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:00.521548 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.521609 (+ 61us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.522677 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.522681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.523580 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.523589 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.524331 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.524337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.524464 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.525091 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:00.525111 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:00.527700 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:00.530415 (+ 2715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:00.530423 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:00.530429 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:00.530855 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:00.531163 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:00.531166 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:00.531168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:00.531285 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:00.531687 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:00.531693 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:00.531830 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.531963 (+ 133us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.532166 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":357,"threads_started":1}
I20260925 15:07:00.532662 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.519612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48996 (local address 127.4.205.254:44325)
0925 15:07:00.520346 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.520350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.520370 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:07:00.520473 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.520476 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.520527 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.520611 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:00.521725 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.522479 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.523724 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.524151 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.524252 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.527851 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:00.527855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:00.527858 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:00.527883 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:00.530302 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:00.530946 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:00.530949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:00.530952 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:00.531062 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:00.531365 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:00.531368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:00.531370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:00.531593 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:00.531692 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.532032 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.532274 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":96,"server-negotiator.queue_time_us":641,"thread_start_us":372,"threads_started":1}
I20260925 15:07:00.537374 6042 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:48996:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:07:00.538475 6042 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:00.544436 6178 tablet_service.cc:1467] Processing CreateTablet for tablet b072549c548b4ffcb56d9495f7e85cf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0681c96875a46d6a60bdfd674b6851a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:00.544772 6316 tablet_service.cc:1467] Processing CreateTablet for tablet b072549c548b4ffcb56d9495f7e85cf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0681c96875a46d6a60bdfd674b6851a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:00.545004 6178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b072549c548b4ffcb56d9495f7e85cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.545258 6316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b072549c548b4ffcb56d9495f7e85cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.547576 6456 tablet_service.cc:1467] Processing CreateTablet for tablet b072549c548b4ffcb56d9495f7e85cf2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f0681c96875a46d6a60bdfd674b6851a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:00.547997 6456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b072549c548b4ffcb56d9495f7e85cf2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:00.552114 6547 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: Bootstrap starting.
I20260925 15:07:00.552131 6546 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: Bootstrap starting.
I20260925 15:07:00.553462 6547 tablet_bootstrap.cc:654] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.554018 6546 tablet_bootstrap.cc:654] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.555469 6586 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: Bootstrap starting.
I20260925 15:07:00.555472 6547 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: No bootstrap required, opened a new log
I20260925 15:07:00.555692 6547 ts_tablet_manager.cc:1397] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:07:00.556461 6546 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: No bootstrap required, opened a new log
I20260925 15:07:00.556422 6547 raft_consensus.cc:348] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.556599 6546 ts_tablet_manager.cc:1397] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:07:00.556661 6547 raft_consensus.cc:374] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.556726 6547 raft_consensus.cc:729] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f3f5e9765364593819cf579babd6928, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.556870 6586 tablet_bootstrap.cc:654] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:00.556964 6547 consensus_queue.cc:260] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.557271 6546 raft_consensus.cc:348] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.557410 6553 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.557508 6546 raft_consensus.cc:374] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.557724 6553 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.557765 6546 raft_consensus.cc:729] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4cf715a7d6d479ca0c6e95c90288ee3, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.557997 6546 consensus_queue.cc:260] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.558144 6547 ts_tablet_manager.cc:1428] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:00.558480 6546 ts_tablet_manager.cc:1428] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:00.558825 6556 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.559068 6556 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.562749 6586 tablet_bootstrap.cc:492] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: No bootstrap required, opened a new log
I20260925 15:07:00.562885 6586 ts_tablet_manager.cc:1397] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 15:07:00.563843 6586 raft_consensus.cc:348] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.564010 6586 raft_consensus.cc:374] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:00.564114 6586 raft_consensus.cc:729] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82061903c4ff459298a3c0e2a0d3da42, State: Initialized, Role: FOLLOWER
I20260925 15:07:00.564332 6586 consensus_queue.cc:260] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.565169 6586 ts_tablet_manager.cc:1428] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:07:00.565081 6590 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.565421 6590 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.741573 6556 raft_consensus.cc:482] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:00.741773 6556 raft_consensus.cc:504] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.742319 6556 leader_election.cc:288] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209), 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847)
I20260925 15:07:00.742851 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b072549c548b4ffcb56d9495f7e85cf2" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42" is_pre_election: true
I20260925 15:07:00.742897 6337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b072549c548b4ffcb56d9495f7e85cf2" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3f5e9765364593819cf579babd6928" is_pre_election: true
I20260925 15:07:00.743091 6337 raft_consensus.cc:2454] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 0.
I20260925 15:07:00.743113 6480 raft_consensus.cc:2454] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 0.
I20260925 15:07:00.743518 6130 leader_election.cc:302] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f3f5e9765364593819cf579babd6928, 82061903c4ff459298a3c0e2a0d3da42, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:07:00.743888 6556 raft_consensus.cc:2781] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:00.744016 6556 raft_consensus.cc:482] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:00.744134 6556 raft_consensus.cc:3037] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.746136 6556 raft_consensus.cc:504] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.746622 6556 leader_election.cc:288] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 election: Requested vote from peers 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209), 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847)
I20260925 15:07:00.747117 6480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b072549c548b4ffcb56d9495f7e85cf2" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82061903c4ff459298a3c0e2a0d3da42"
I20260925 15:07:00.747135 6337 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b072549c548b4ffcb56d9495f7e85cf2" candidate_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f3f5e9765364593819cf579babd6928"
I20260925 15:07:00.747331 6337 raft_consensus.cc:3037] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.747354 6480 raft_consensus.cc:3037] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:00.750414 6337 raft_consensus.cc:2454] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 1.
I20260925 15:07:00.750399 6480 raft_consensus.cc:2454] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b4cf715a7d6d479ca0c6e95c90288ee3 in term 1.
I20260925 15:07:00.750833 6129 leader_election.cc:302] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f3f5e9765364593819cf579babd6928, 82061903c4ff459298a3c0e2a0d3da42, b4cf715a7d6d479ca0c6e95c90288ee3; no voters:
I20260925 15:07:00.751132 6556 raft_consensus.cc:2781] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:00.751525 6556 raft_consensus.cc:686] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 LEADER]: Becoming Leader. State: Replica: b4cf715a7d6d479ca0c6e95c90288ee3, State: Running, Role: LEADER
I20260925 15:07:00.751762 6556 consensus_queue.cc:237] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } }
I20260925 15:07:00.753278 6591 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: TxnStatusTablet state changed. Reason: New leader b4cf715a7d6d479ca0c6e95c90288ee3. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.753713 6591 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:00.754374 6593 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:07:00.755038 6593 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:07:00.755149 6041 catalog_manager.cc:5495] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:00.796705 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.791435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.4.205.193:40085)
0925 15:07:00.791610 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.791614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.791629 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:00.791886 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.791889 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.791953 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.792025 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:00.792413 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.793269 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.795148 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.795584 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.795680 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.796110 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.796209 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.796273 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.796436 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.796490 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:07:00.796857 6584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.791300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40085 (local address 127.0.0.1:43580)
0925 15:07:00.791538 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.791558 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.791695 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.792072 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.792077 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.792103 (+ 26us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:00.792276 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.792287 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.793392 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.793399 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.795016 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.795030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.795713 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.795719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.795916 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.795980 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.796310 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.796327 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.796406 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.796547 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:07:00.800570 6480 raft_consensus.cc:1261] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Refusing update from remote peer b4cf715a7d6d479ca0c6e95c90288ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:00.800766 6337 raft_consensus.cc:1261] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Refusing update from remote peer b4cf715a7d6d479ca0c6e95c90288ee3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:00.801208 6556 consensus_queue.cc:1035] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:00.801518 6591 consensus_queue.cc:1035] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:00.810952 6553 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: TxnStatusTablet state changed. Reason: New leader b4cf715a7d6d479ca0c6e95c90288ee3. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.811236 6553 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.814512 6590 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: TxnStatusTablet state changed. Reason: New leader b4cf715a7d6d479ca0c6e95c90288ee3. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.814839 6590 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.818051 6553 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.818354 6553 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 6f3f5e9765364593819cf579babd6928: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.825328 6556 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.825618 6556 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:00.828398 6590 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.828625 6590 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P 82061903c4ff459298a3c0e2a0d3da42: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:00.835541 6591 tablet_replica.cc:439] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 } } peers { permanent_uuid: "6f3f5e9765364593819cf579babd6928" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44209 } } peers { permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 } } }
I20260925 15:07:00.835886 6591 tablet_replica.cc:442] T b072549c548b4ffcb56d9495f7e85cf2 P b4cf715a7d6d479ca0c6e95c90288ee3: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:00.843267 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.833262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.4.205.194:44209)
0925 15:07:00.836587 (+ 3325us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.836593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.836977 (+ 384us) server_negotiation.cc:407] Connection header received
0925 15:07:00.837183 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.837188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.837289 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.837446 (+ 157us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:00.837902 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.839479 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.840834 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.841493 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.841647 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.842003 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.842703 (+ 700us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.842838 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.842881 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.842953 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3192,"thread_start_us":3080,"threads_started":1}
I20260925 15:07:00.857259 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.852398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:37927 (local address 127.4.205.193:40085)
0925 15:07:00.852591 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.852599 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.852919 (+ 320us) server_negotiation.cc:407] Connection header received
0925 15:07:00.853103 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.853107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.853163 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.853262 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:00.853701 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.854462 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.855712 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.856125 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.856259 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.856601 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.856716 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.856801 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.856902 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.856972 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:07:00.857362 6610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.852296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40085 (local address 127.4.205.194:37927)
0925 15:07:00.852831 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.852845 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.852983 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.853276 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.853281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.853291 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:00.853533 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.853539 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.854588 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.854596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.855584 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.855594 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.856276 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.856281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.856403 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.856442 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.856802 (+ 360us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.856813 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.856886 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.857007 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":287,"threads_started":1}
I20260925 15:07:00.865463 6476 raft_consensus.cc:1261] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Refusing update from remote peer 6f3f5e9765364593819cf579babd6928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:00.865883 6202 raft_consensus.cc:1261] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Refusing update from remote peer 6f3f5e9765364593819cf579babd6928: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:00.866091 6577 consensus_queue.cc:1035] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82061903c4ff459298a3c0e2a0d3da42" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:00.866454 6553 consensus_queue.cc:1035] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b4cf715a7d6d479ca0c6e95c90288ee3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:00.904713 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.898543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50371 (local address 127.4.205.194:44209)
0925 15:07:00.898702 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.898708 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.899416 (+ 708us) server_negotiation.cc:407] Connection header received
0925 15:07:00.899486 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.899490 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.899554 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.899639 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:00.900095 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.900914 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.902405 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.903053 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.903198 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.903785 (+ 587us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.903895 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.904049 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.904375 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.904458 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:07:00.905217 6616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.898573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44209 (local address 127.4.205.193:50371)
0925 15:07:00.899184 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:00.899202 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:00.899325 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:00.899666 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:00.899671 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:00.899681 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:00.899970 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.899978 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.901016 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.901022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.902251 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:00.902261 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.903349 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.903356 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:00.903534 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.903598 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:00.904050 (+ 452us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:00.904063 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:07:00.904162 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:07:00.904911 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260925 15:07:00.941025 6041 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:00.941347 6041 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48988:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:00.944993 6041 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5878004d445745beb2111d52bfe1dc57: Sending DeleteTablet for 3 replicas of tablet e8550aefc92e4095bacacf0d6fa0cfcb
I20260925 15:07:00.946138 6456 tablet_service.cc:1514] Processing DeleteTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:00 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49672
I20260925 15:07:00.946677 6316 tablet_service.cc:1514] Processing DeleteTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:00 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:57174
I20260925 15:07:00.945696 6178 tablet_service.cc:1514] Processing DeleteTablet for tablet e8550aefc92e4095bacacf0d6fa0cfcb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:00 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:43574
I20260925 15:07:00.960353 6622 tablet_replica.cc:330] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: stopping tablet replica
I20260925 15:07:00.962347 6622 raft_consensus.cc:2229] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:00.962668 6622 raft_consensus.cc:2258] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.963726 6622 ts_tablet_manager.cc:1905] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.973747 6619 tablet_replica.cc:330] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: stopping tablet replica
I20260925 15:07:00.973960 6619 raft_consensus.cc:2229] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:00.974222 6619 raft_consensus.cc:2258] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.975311 6619 ts_tablet_manager.cc:1905] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.975715 6620 tablet_replica.cc:330] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: stopping tablet replica
I20260925 15:07:00.975942 6620 raft_consensus.cc:2229] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:00.976501 6620 raft_consensus.cc:2258] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:00.977450 6620 ts_tablet_manager.cc:1905] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:00.977985 6622 ts_tablet_manager.cc:1918] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:00.981241 6622 log.cc:1196] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/e8550aefc92e4095bacacf0d6fa0cfcb
I20260925 15:07:00.981873 6622 ts_tablet_manager.cc:1939] T e8550aefc92e4095bacacf0d6fa0cfcb P b4cf715a7d6d479ca0c6e95c90288ee3: Deleting consensus metadata
I20260925 15:07:00.982939 6619 ts_tablet_manager.cc:1918] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:00.983059 6619 log.cc:1196] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/e8550aefc92e4095bacacf0d6fa0cfcb
I20260925 15:07:00.983574 6619 ts_tablet_manager.cc:1939] T e8550aefc92e4095bacacf0d6fa0cfcb P 82061903c4ff459298a3c0e2a0d3da42: Deleting consensus metadata
I20260925 15:07:00.984553 6026 catalog_manager.cc:4841] TS 82061903c4ff459298a3c0e2a0d3da42 (127.4.205.195:46847): tablet e8550aefc92e4095bacacf0d6fa0cfcb (table test-table [id=aeafa150731a4d7e827e303c2682cf9c]) successfully deleted
I20260925 15:07:00.984939 6029 catalog_manager.cc:4841] TS b4cf715a7d6d479ca0c6e95c90288ee3 (127.4.205.193:40085): tablet e8550aefc92e4095bacacf0d6fa0cfcb (table test-table [id=aeafa150731a4d7e827e303c2682cf9c]) successfully deleted
I20260925 15:07:00.987818 6620 ts_tablet_manager.cc:1918] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:00.987954 6620 log.cc:1196] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/e8550aefc92e4095bacacf0d6fa0cfcb
Sep 25 15:07:00 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:00.988401 6620 ts_tablet_manager.cc:1939] T e8550aefc92e4095bacacf0d6fa0cfcb P 6f3f5e9765364593819cf579babd6928: Deleting consensus metadata
I20260925 15:07:00.989478 6027 catalog_manager.cc:4841] TS 6f3f5e9765364593819cf579babd6928 (127.4.205.194:44209): tablet e8550aefc92e4095bacacf0d6fa0cfcb (table test-table [id=aeafa150731a4d7e827e303c2682cf9c]) successfully deleted
I20260925 15:07:00.997059 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:00.961497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43586 (local address 127.4.205.193:40085)
0925 15:07:00.961692 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:00.961698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:00.972898 (+ 11200us) server_negotiation.cc:407] Connection header received
0925 15:07:00.973318 (+ 420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:00.973323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:00.973414 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:00.973492 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:00.978171 (+ 4679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.979305 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.982820 (+ 3515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:00.983486 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:00.983588 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:00.989545 (+ 5957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:00.989565 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:00.989584 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:00.989670 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:00.993937 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:00.994781 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:00.994790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:00.994795 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:00.994929 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:00.995312 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:00.995317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:00.995320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:00.995742 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:00.996025 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:00.996319 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:07:00.996532 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:07:00.997810 6243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43586
I20260925 15:07:01.016671 6585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:01.001259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49000 (local address 127.4.205.254:44325)
0925 15:07:01.001456 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:01.001462 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:01.001866 (+ 404us) server_negotiation.cc:407] Connection header received
0925 15:07:01.002021 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:01.002026 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:01.002107 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:01.002230 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:01.003820 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.005008 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.006731 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.007382 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.007518 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:01.010244 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:01.010251 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:01.010256 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:01.010292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:01.013877 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.014594 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.014601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.014606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.014743 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.015141 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.015146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.015149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.015523 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:01.015749 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:01.016110 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 15:07:01.016329 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 15:07:01.017529 6041 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49000
I20260925 15:07:01.019567 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6113
I20260925 15:07:01.037128 6544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:01.021281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43596 (local address 127.4.205.193:40085)
0925 15:07:01.021454 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:01.021459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:01.021830 (+ 371us) server_negotiation.cc:407] Connection header received
0925 15:07:01.022057 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:01.022063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:01.022146 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:01.022646 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:01.023901 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.025093 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.026703 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.027453 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.027580 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:01.030164 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:01.030170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:01.030175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:01.030209 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:01.034250 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.034986 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.034992 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.034996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.035167 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.035563 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.035568 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.035572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.035914 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:01.036187 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:01.036464 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:07:01.036680 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 15:07:01.038166 6243 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43596
W20260925 15:07:01.041216 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6113: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:01.041417 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6113
I20260925 15:07:01.064797 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6252
Sep 25 15:07:01 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:01.079592 6607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:01.065561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57198 (local address 127.4.205.194:44209)
0925 15:07:01.065789 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:01.065797 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:01.065852 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:07:01.066061 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:01.066065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:01.066148 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:01.066272 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:01.067372 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.068100 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.069278 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.069848 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.069988 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:01.073930 (+ 3942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:01.073942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:01.073956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:01.073985 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:01.076858 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.077753 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.077760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.077762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.077886 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.078256 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.078259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.078261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.078588 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:01.078839 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:01.079127 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:07:01.079271 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:07:01.080312 6382 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57198
W20260925 15:07:01.080952 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6252: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:01.081135 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6252
I20260925 15:07:01.099855 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6391
Sep 25 15:07:01 dist-test-slave-64fx krb5kdc[5997](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348819, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:01.114140 6627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:01.100531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.4.205.195:46847)
0925 15:07:01.100988 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:01.100993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:01.101007 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:07:01.101055 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:01.101058 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:01.101122 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:01.101241 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:01.102445 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.103230 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.104450 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:01.104849 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:01.104977 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:01.108819 (+ 3842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:01.108832 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:01.108847 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:01.108876 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:01.111385 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.112002 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.112028 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.112030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.112149 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:01.112534 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:01.112541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:01.112545 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:01.112938 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:01.113232 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:01.113520 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:07:01.113869 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":217,"threads_started":1}
W20260925 15:07:01.114734 6521 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49680
W20260925 15:07:01.115327 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6391: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:01.115465 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6391
I20260925 15:07:01.131394 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6010
I20260925 15:07:01.132361 6082 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:01.240717 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6010
2026-09-25T15:07:01Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4817 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:01 dist-test-slave-64fx krb5kdc[6631](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:01 dist-test-slave-64fx krb5kdc[6631](info): set up 2 sockets
Sep 25 15:07:01 dist-test-slave-64fx krb5kdc[6631](info): commencing operation
krb5kdc: starting...
W20260925 15:07:02.277412 4919 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.005s sys 0.009s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:02 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:02.445060 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42379
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:42379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:02.741943 6644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:02.742238 6644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:02.742394 6644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:02.748868 6644 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:02.748980 6644 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:07:02.749063 6644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:02.749177 6644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:02.749250 6644 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:02.749331 6644 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:02.757967 6644 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:42379
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:02.759017 6644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:02.760195 6644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:02.771436 6649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:02.771530 6652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:02.771579 6650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:02.772298 6644 server_base.cc:1034] running on GCE node
I20260925 15:07:02.773211 6644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:02.774518 6644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:02.775761 6644 hybrid_clock.cc:648] HybridClock initialized: now 1790348822775688 us; error 50 us; skew 500 ppm
I20260925 15:07:02.776073 6644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:02 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348822, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:02.783203 6644 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:02.784427 6644 webserver.cc:460] Webserver started at http://127.4.205.254:35913/ using document root <none> and password file <none>
I20260925 15:07:02.785008 6644 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:02.785086 6644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:02.785324 6644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:02.787918 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "f0e0b47f88a848e19a0c45c585b83ba6"
format_stamp: "Formatted at 2026-09-25 15:07:02 on dist-test-slave-64fx"
server_key: "8a1496e187613ca5a1ead2ac740d9b20"
server_key_iv: "63d05ab6a93583afc173f99f90b21e6d"
server_key_version: "encryptionkey@0"
I20260925 15:07:02.788594 6644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "f0e0b47f88a848e19a0c45c585b83ba6"
format_stamp: "Formatted at 2026-09-25 15:07:02 on dist-test-slave-64fx"
server_key: "8a1496e187613ca5a1ead2ac740d9b20"
server_key_iv: "63d05ab6a93583afc173f99f90b21e6d"
server_key_version: "encryptionkey@0"
I20260925 15:07:02.792928 6644 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 15:07:02.796031 6659 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:02.797112 6644 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:02.797314 6644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "f0e0b47f88a848e19a0c45c585b83ba6"
format_stamp: "Formatted at 2026-09-25 15:07:02 on dist-test-slave-64fx"
server_key: "8a1496e187613ca5a1ead2ac740d9b20"
server_key_iv: "63d05ab6a93583afc173f99f90b21e6d"
server_key_version: "encryptionkey@0"
I20260925 15:07:02.797495 6644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:02.811203 6644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:02.814747 6644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:02.814993 6644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:02.835018 6644 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:42379
I20260925 15:07:02.835036 6720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:42379 every 8 connection(s)
I20260925 15:07:02.836234 6644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:02.837952 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6644
I20260925 15:07:02.838158 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:02.838744 4919 external_mini_cluster.cc:1428] Setting key a03ebccbad4b168f8bc0f8865e27b10a
I20260925 15:07:02.839913 6721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:07:02 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:02.852464 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Bootstrap starting.
I20260925 15:07:02.856139 6721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:02.857239 6721 log.cc:824] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:02.859860 6721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: No bootstrap required, opened a new log
I20260925 15:07:02.860371 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:02.840639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.4.205.254:42379)
0925 15:07:02.841373 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:02.841388 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:02.841432 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:07:02.843155 (+ 1723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:02.843179 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:02.843642 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:02.844030 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:02.845445 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:02.847441 (+ 1996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:02.848549 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:02.849485 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:02.849684 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:02.853130 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:02.853139 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:02.853154 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:02.853193 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:02.856770 (+ 3577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:02.857327 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:02.857337 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:02.857345 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:02.857522 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:02.857880 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:02.857887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:02.857894 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:02.858382 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:02.858686 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:02.859094 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 15:07:02.859455 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":303,"threads_started":1}
I20260925 15:07:02.866892 6721 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } }
I20260925 15:07:02.867233 6721 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:02.867327 6721 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e0b47f88a848e19a0c45c585b83ba6, State: Initialized, Role: FOLLOWER
I20260925 15:07:02.867792 6721 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } }
I20260925 15:07:02.867990 6721 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:02.868117 6721 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:02.868281 6721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:02.870442 6721 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } }
I20260925 15:07:02.870832 6721 leader_election.cc:302] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e0b47f88a848e19a0c45c585b83ba6; no voters:
I20260925 15:07:02.871244 6721 leader_election.cc:288] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:02.871331 6726 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:02.871982 6726 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [term 1 LEADER]: Becoming Leader. State: Replica: f0e0b47f88a848e19a0c45c585b83ba6, State: Running, Role: LEADER
I20260925 15:07:02.872521 6726 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } }
I20260925 15:07:02.872949 6721 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:02.874919 6727 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } } }
I20260925 15:07:02.875222 6727 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:02.875941 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e0b47f88a848e19a0c45c585b83ba6. Latest consensus state: current_term: 1 leader_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e0b47f88a848e19a0c45c585b83ba6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42379 } } }
I20260925 15:07:02.876914 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:02.876046 6731 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:02.881944 6731 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:02.889616 6731 catalog_manager.cc:1347] Generated new cluster ID: a427277070854ddd90391bd8e53a4547
I20260925 15:07:02.889709 6731 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:02.929028 6731 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:02.930804 6731 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:02.952925 6731 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Generated new TSK 0
I20260925 15:07:02.953841 6731 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:03.003338 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42379
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:03.304316 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:03.304567 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:03.304711 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:03.311134 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:03.311234 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:03.311323 6747 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:07:03.311404 6747 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:03.311507 6747 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:03.320416 6747 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:03.321551 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:03.322767 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:03.334818 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:03.334877 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:03.334898 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:03.335744 6747 server_base.cc:1034] running on GCE node
I20260925 15:07:03.336293 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:03.337146 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:03.338326 6747 hybrid_clock.cc:648] HybridClock initialized: now 1790348823338292 us; error 39 us; skew 500 ppm
I20260925 15:07:03.338629 6747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:03 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:03.341518 6747 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:03.342846 6747 webserver.cc:460] Webserver started at http://127.4.205.193:37353/ using document root <none> and password file <none>
I20260925 15:07:03.343468 6747 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:03.343600 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:03.343892 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:03.346702 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "e987a49e415e455380387cf2beca770d"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "9b61507f7974e12f1fbbfdc2927587da"
server_key_iv: "4dfc937b5cddbfaed9556389ab709e80"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.347489 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "e987a49e415e455380387cf2beca770d"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "9b61507f7974e12f1fbbfdc2927587da"
server_key_iv: "4dfc937b5cddbfaed9556389ab709e80"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.352042 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:03.355120 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:03.356240 6747 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:03.356392 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "e987a49e415e455380387cf2beca770d"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "9b61507f7974e12f1fbbfdc2927587da"
server_key_iv: "4dfc937b5cddbfaed9556389ab709e80"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.356571 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:03.383129 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:03.387090 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:03.387403 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:03.389714 6747 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:03.389844 6747 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:03.389959 6747 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:03.390043 6747 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:03 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:03.410919 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.390290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37463 (local address 127.4.205.254:42379)
0925 15:07:03.390906 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:03.390912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:03.391852 (+ 940us) server_negotiation.cc:407] Connection header received
0925 15:07:03.393645 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:03.393654 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:03.393758 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:03.393874 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:03.395931 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.396992 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.399583 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.400003 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.400125 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.404262 (+ 4137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:03.404266 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:03.404269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:03.404298 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:03.407192 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.408111 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.408118 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.408121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.408275 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.408954 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.408960 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.408964 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.409316 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:03.409467 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:03.410476 (+ 1009us) server_negotiation.cc:299] Negotiation successful
0925 15:07:03.410629 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":397,"threads_started":1}
I20260925 15:07:03.412883 6772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.390899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.193:37463)
0925 15:07:03.391698 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:03.391739 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:03.393367 (+ 1628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:03.394141 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:03.394153 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:03.394863 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:03.395684 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.395715 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.397132 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.397137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.399376 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.399387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.400247 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.400255 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.400442 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.401124 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:03.401164 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:03.404006 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:03.407319 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.407330 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.407350 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.407973 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.408433 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.408443 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.408448 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.408763 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.409475 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:03.409487 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:03.410048 (+ 561us) client_negotiation.cc:769] Sending connection context
0925 15:07:03.410429 (+ 381us) client_negotiation.cc:240] Negotiation successful
0925 15:07:03.411117 (+ 688us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":351,"threads_started":1}
I20260925 15:07:03.431619 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.418124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57145 (local address 127.4.205.254:42379)
0925 15:07:03.418293 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:03.418298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:03.419253 (+ 955us) server_negotiation.cc:407] Connection header received
0925 15:07:03.419423 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:03.419429 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:03.419508 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:03.419598 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:03.421024 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.421781 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.423471 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.423865 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.423977 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.425585 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:03.425592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:03.425598 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:03.425651 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:03.428442 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.429340 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.429348 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.429352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.429510 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.430043 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.430050 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.430056 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.430385 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:03.430515 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:03.431050 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 15:07:03.431296 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:03.431764 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.418424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.193:57145)
0925 15:07:03.419156 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:03.419170 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:03.419307 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:03.419647 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:03.419658 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:03.420376 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:03.420870 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.420885 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.421934 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.421941 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.423337 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.423351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.424033 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.424041 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.424186 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.424782 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:03.424797 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:03.425412 (+ 615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:03.428620 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.428626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.428630 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.429205 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.429710 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.429720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.429726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.429912 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.430506 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:03.430510 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:03.430755 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 15:07:03.430906 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:07:03.431289 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":481,"threads_started":1}
I20260925 15:07:03.439356 6747 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:46401
I20260925 15:07:03.439394 6882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:46401 every 8 connection(s)
I20260925 15:07:03.440454 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:03.442226 6883 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42379
I20260925 15:07:03.442430 6883 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:03.442989 6883 heartbeater.cc:511] Master 127.4.205.254:42379 requested a full tablet report, sending...
I20260925 15:07:03.444707 6676 ts_manager.cc:194] Registered new tserver with Master: e987a49e415e455380387cf2beca770d (127.4.205.193:46401)
I20260925 15:07:03.446219 6676 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:37463
I20260925 15:07:03.448469 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6747
I20260925 15:07:03.448688 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:03.449278 4919 external_mini_cluster.cc:1428] Setting key b14b7a55535ecb053591d7e8b85fadf0
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:03.485507 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42379
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:03.793809 6887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:03.794041 6887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:03.794216 6887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:03.801141 6887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:03.801270 6887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:03.801430 6887 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:07:03.801505 6887 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:03.801654 6887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:03.810707 6887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:03.811694 6887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:03.812839 6887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:03.825048 6892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:03.825086 6893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:03.825110 6895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:03.825874 6887 server_base.cc:1034] running on GCE node
I20260925 15:07:03.826411 6887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:03.827219 6887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:03.828400 6887 hybrid_clock.cc:648] HybridClock initialized: now 1790348823828346 us; error 54 us; skew 500 ppm
I20260925 15:07:03.828699 6887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:03 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:03.831717 6887 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:03.832978 6887 webserver.cc:460] Webserver started at http://127.4.205.194:45225/ using document root <none> and password file <none>
I20260925 15:07:03.833657 6887 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:03.833794 6887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:03.834084 6887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:03.836686 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "fe1169b9bc37eedc9231623f3fc98d63"
server_key_iv: "2ff2149ec5c2f51e2c1a7a0b0b830ab5"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.837409 6887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "fe1169b9bc37eedc9231623f3fc98d63"
server_key_iv: "2ff2149ec5c2f51e2c1a7a0b0b830ab5"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.842195 6887 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:07:03.845389 6902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:03.846602 6887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:03.846768 6887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
format_stamp: "Formatted at 2026-09-25 15:07:03 on dist-test-slave-64fx"
server_key: "fe1169b9bc37eedc9231623f3fc98d63"
server_key_iv: "2ff2149ec5c2f51e2c1a7a0b0b830ab5"
server_key_version: "encryptionkey@0"
I20260925 15:07:03.846949 6887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:03.869477 6887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:03.873500 6887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:03.873807 6887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:03.876073 6887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:03.876192 6887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:03.876327 6887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:03.876431 6887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:03 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348823, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:03.897878 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.876646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:42829 (local address 127.4.205.254:42379)
0925 15:07:03.876830 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:03.876836 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:03.878195 (+ 1359us) server_negotiation.cc:407] Connection header received
0925 15:07:03.880333 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:03.880339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:03.880421 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:03.880543 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:03.882639 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.884048 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.885619 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.886359 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.886515 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.890785 (+ 4270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:03.890795 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:03.890803 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:03.890841 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:03.894139 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.895138 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.895146 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.895152 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.895298 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.895914 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.895922 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.895928 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.896376 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:03.896558 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:03.897126 (+ 568us) server_negotiation.cc:299] Negotiation successful
0925 15:07:03.897418 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:07:03.898435 6911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.877197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.194:42829)
0925 15:07:03.878012 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:03.878059 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:03.880032 (+ 1973us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:03.880779 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:03.880792 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:03.881616 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:03.882440 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.882457 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.884212 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.884216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.885463 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.885473 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.886617 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.886630 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.886834 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.887506 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:03.887525 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:03.890521 (+ 2996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:03.894286 (+ 3765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.894291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.894299 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.894945 (+ 646us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.895428 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.895439 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.895443 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.895758 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.896524 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:03.896529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:03.896774 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 15:07:03.896965 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:07:03.897255 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":378,"threads_started":1}
I20260925 15:07:03.917230 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.903362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:52765 (local address 127.4.205.254:42379)
0925 15:07:03.903556 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:03.903564 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:03.904078 (+ 514us) server_negotiation.cc:407] Connection header received
0925 15:07:03.904198 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:03.904203 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:03.904284 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:03.904389 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:03.905773 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.906788 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.908958 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.909365 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.909490 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.911766 (+ 2276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:03.911774 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:03.911781 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:03.911817 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:03.914864 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.915472 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.915479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.915484 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.915679 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:03.915987 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:03.915992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:03.915996 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:03.916248 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:03.916412 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:03.916750 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:07:03.916930 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:07:03.917337 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:03.903363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.194:52765)
0925 15:07:03.903987 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:03.904005 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:03.904104 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:03.904408 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:03.904411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:03.905040 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:03.905530 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.905548 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.906928 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.906934 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.908807 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:03.908825 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:03.909539 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:03.909544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:03.909773 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:03.910493 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:03.910512 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:03.911526 (+ 1014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:03.914978 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.914986 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.914990 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.915357 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.915764 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:03.915770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:03.915774 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:03.915890 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:03.916374 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:03.916379 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:03.916552 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:07:03.916676 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:07:03.916934 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":350,"threads_started":1}
I20260925 15:07:03.929371 6887 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:35255
I20260925 15:07:03.929410 7021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:35255 every 8 connection(s)
I20260925 15:07:03.930420 6887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:03.932085 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 6887
I20260925 15:07:03.932278 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:03.932332 7022 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42379
I20260925 15:07:03.932533 7022 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:03.932828 4919 external_mini_cluster.cc:1428] Setting key d43b4393961dc4f6b81b481515e3a749
I20260925 15:07:03.933205 7022 heartbeater.cc:511] Master 127.4.205.254:42379 requested a full tablet report, sending...
I20260925 15:07:03.934551 6675 ts_manager.cc:194] Registered new tserver with Master: ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255)
I20260925 15:07:03.935755 6675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:42829
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:03.970674 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42379
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39091
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:04.276895 7026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:04.277186 7026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:04.277379 7026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:04.284353 7026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:04.284480 7026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:04.284642 7026 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 15:07:04.284724 7026 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:04.284857 7026 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:04.294080 7026 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39091
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:42379
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:04.295262 7026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:04.296523 7026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:04.309062 7032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:04.309128 7034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:04.309091 7031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:04.309836 7026 server_base.cc:1034] running on GCE node
I20260925 15:07:04.310453 7026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:04.311353 7026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:04.312568 7026 hybrid_clock.cc:648] HybridClock initialized: now 1790348824312528 us; error 48 us; skew 500 ppm
I20260925 15:07:04.312863 7026 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:04 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348824, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:04.316017 7026 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:04.317255 7026 webserver.cc:460] Webserver started at http://127.4.205.195:41407/ using document root <none> and password file <none>
I20260925 15:07:04.317910 7026 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:04.318007 7026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:04.318261 7026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:04.320889 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "f40f0a8ef4614255b66b08055eec0113"
format_stamp: "Formatted at 2026-09-25 15:07:04 on dist-test-slave-64fx"
server_key: "e65231a0e49393992a3be7671d8e6bc9"
server_key_iv: "ba5fb1efae1ffab682479c2c3a5b8eb6"
server_key_version: "encryptionkey@0"
I20260925 15:07:04.321633 7026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "f40f0a8ef4614255b66b08055eec0113"
format_stamp: "Formatted at 2026-09-25 15:07:04 on dist-test-slave-64fx"
server_key: "e65231a0e49393992a3be7671d8e6bc9"
server_key_iv: "ba5fb1efae1ffab682479c2c3a5b8eb6"
server_key_version: "encryptionkey@0"
I20260925 15:07:04.326355 7026 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:07:04.329407 7041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:04.330667 7026 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:04.330852 7026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "f40f0a8ef4614255b66b08055eec0113"
format_stamp: "Formatted at 2026-09-25 15:07:04 on dist-test-slave-64fx"
server_key: "e65231a0e49393992a3be7671d8e6bc9"
server_key_iv: "ba5fb1efae1ffab682479c2c3a5b8eb6"
server_key_version: "encryptionkey@0"
I20260925 15:07:04.331051 7026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:04.350488 7026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:04.353968 7026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:04.354228 7026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:04.356577 7026 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:04.356709 7026 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:04.356858 7026 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:04.356945 7026 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:04 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348824, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:04.382043 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.357348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:57129 (local address 127.4.205.254:42379)
0925 15:07:04.357510 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.357515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.358954 (+ 1439us) server_negotiation.cc:407] Connection header received
0925 15:07:04.362060 (+ 3106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.362069 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.362169 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.362294 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:04.364697 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.365667 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.368647 (+ 2980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.369235 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.369405 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.374235 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:04.374248 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:04.374252 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:04.374322 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:04.378253 (+ 3931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.378900 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.378908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.378914 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.379110 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.379726 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.379767 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.379774 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.380185 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:04.380339 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.381217 (+ 878us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.381514 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:07:04.382778 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.357936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.195:57129)
0925 15:07:04.358789 (+ 853us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.358832 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.361711 (+ 2879us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.362592 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.362605 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.363513 (+ 908us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:04.364479 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.364534 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.365858 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.365866 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.368457 (+ 2591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.368472 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.369528 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.369541 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.369788 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.370515 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:04.370552 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:04.374012 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:04.378381 (+ 4369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.378387 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.378397 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.378761 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.379210 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.379216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.379221 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.379562 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.380330 (+ 768us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:04.380348 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:04.380804 (+ 456us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.381116 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.381543 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":319,"threads_started":1}
I20260925 15:07:04.408628 7026 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:37967
I20260925 15:07:04.408645 7160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:37967 every 8 connection(s)
I20260925 15:07:04.409983 7026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:04.410432 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.389128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:42435 (local address 127.4.205.254:42379)
0925 15:07:04.389301 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.389305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.389619 (+ 314us) server_negotiation.cc:407] Connection header received
0925 15:07:04.389781 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.389788 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.389880 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.389977 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:04.391219 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.392044 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.400128 (+ 8084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.400889 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.401024 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.403607 (+ 2583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:04.403613 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:04.403618 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:04.403657 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:04.407723 (+ 4066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.408481 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.408487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.408492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.408631 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.409007 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.409013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.409017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.409270 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:04.409430 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.409867 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.410080 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:07:04.410563 7124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.388761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.4.205.195:42435)
0925 15:07:04.389495 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.389513 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.389659 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.390028 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.390036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.390677 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:04.391082 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.391096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.392193 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.392203 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.399924 (+ 7721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.399944 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.401150 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.401171 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.401387 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.402240 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:04.402268 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:04.403420 (+ 1152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:04.407887 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.407894 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.407900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.408323 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.408735 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.408742 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.408750 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.408890 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.409384 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:04.409391 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:04.409600 (+ 209us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.409758 (+ 158us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.410098 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":537,"threads_started":1}
I20260925 15:07:04.412366 7161 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42379
I20260925 15:07:04.412541 7161 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:04.413141 7161 heartbeater.cc:511] Master 127.4.205.254:42379 requested a full tablet report, sending...
I20260925 15:07:04.414644 6675 ts_manager.cc:194] Registered new tserver with Master: f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.415431 6675 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:57129
I20260925 15:07:04.415580 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7026
I20260925 15:07:04.415710 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:04.416290 4919 external_mini_cluster.cc:1428] Setting key cc781b8aceb9b9b30011cd4d37a441e3
I20260925 15:07:04.418988 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:07:04 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260925 15:07:04.448794 6883 heartbeater.cc:503] Master 127.4.205.254:42379 was elected leader, sending a full tablet report...
Sep 25 15:07:04 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:04.464916 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.452581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54886 (local address 127.4.205.254:42379)
0925 15:07:04.452773 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.452777 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.453033 (+ 256us) server_negotiation.cc:407] Connection header received
0925 15:07:04.453160 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.453164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.453220 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.453296 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:04.454589 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.455296 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.456436 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.456820 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.456903 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.460380 (+ 3477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:04.460398 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:04.460401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:04.460444 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:04.462814 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.463475 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.463479 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.463481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.463584 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.463875 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.463878 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.463879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.464107 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:04.464302 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.464517 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.464657 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:04.468420 6675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:04.474910 6675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:04.497166 7178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.488257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.0.0.1:58624)
0925 15:07:04.490250 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.490265 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.490383 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.490937 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.490941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.490961 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.491410 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.491422 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.493034 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.493039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.495128 (+ 2089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.495137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.496244 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.496249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.496413 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.496472 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.496610 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.496768 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1905,"mutex_wait_us":5,"thread_start_us":1801,"threads_started":1}
I20260925 15:07:04.497331 7174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.486254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:37967 (local address 127.0.0.1:39978)
0925 15:07:04.489763 (+ 3509us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.489797 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.489925 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.490688 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.490694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.490728 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.491086 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.491100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.492624 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.492632 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.494886 (+ 2254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.494900 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.496624 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.496629 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.496772 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.496797 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.496900 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.497005 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3389,"thread_start_us":1853,"threads_started":1}
I20260925 15:07:04.497749 7176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.485899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:46401 (local address 127.0.0.1:33802)
0925 15:07:04.487960 (+ 2061us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.487995 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.488167 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.491196 (+ 3029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.491201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.491230 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.491646 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.491659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.493227 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.493240 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.495182 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.495196 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.497003 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.497008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.497188 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.497220 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.497328 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.497422 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1848,"mutex_wait_us":1,"thread_start_us":3590,"threads_started":1}
I20260925 15:07:04.498010 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.488774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.4.205.194:35255)
0925 15:07:04.489353 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.489362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.490357 (+ 995us) server_negotiation.cc:407] Connection header received
0925 15:07:04.490548 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.490554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.490763 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.490952 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.491547 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.492906 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.495263 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.496073 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.496219 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.496403 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.497541 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.497731 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":259,"threads_started":1}
I20260925 15:07:04.498299 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.486667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.4.205.195:37967)
0925 15:07:04.487293 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.487301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.489901 (+ 2600us) server_negotiation.cc:407] Connection header received
0925 15:07:04.490056 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.490066 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.490295 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.490465 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.491328 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.492501 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.495046 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.496442 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.496560 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.496780 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.496940 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.497083 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466,"thread_start_us":693,"threads_started":1}
I20260925 15:07:04.499220 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.486070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33802 (local address 127.4.205.193:46401)
0925 15:07:04.486598 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.486607 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.490444 (+ 3837us) server_negotiation.cc:407] Connection header received
0925 15:07:04.490569 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.490576 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.491019 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.491190 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.491831 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.493110 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.495365 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.496607 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.496710 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.496874 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.498026 (+ 1152us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.498942 (+ 916us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":790,"threads_started":1}
I20260925 15:07:04.501401 6952 tablet_service.cc:1467] Processing CreateTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 (DEFAULT_TABLE table=test-table [id=2c7ca41558e349b680037eb28a775ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.501400 7091 tablet_service.cc:1467] Processing CreateTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 (DEFAULT_TABLE table=test-table [id=2c7ca41558e349b680037eb28a775ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.502069 6813 tablet_service.cc:1467] Processing CreateTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 (DEFAULT_TABLE table=test-table [id=2c7ca41558e349b680037eb28a775ed3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.502789 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2793ad9ef704967b290de1ebfd7b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.502789 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2793ad9ef704967b290de1ebfd7b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.503895 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2793ad9ef704967b290de1ebfd7b6e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.513548 7180 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Bootstrap starting.
I20260925 15:07:04.516542 7181 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Bootstrap starting.
I20260925 15:07:04.517642 7180 tablet_bootstrap.cc:654] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.518878 7180 log.cc:824] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:04.519783 7181 tablet_bootstrap.cc:654] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.520959 7181 log.cc:824] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:04.521763 7180 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: No bootstrap required, opened a new log
I20260925 15:07:04.522012 7180 ts_tablet_manager.cc:1397] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:07:04.523600 7182 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Bootstrap starting.
I20260925 15:07:04.523741 7181 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: No bootstrap required, opened a new log
I20260925 15:07:04.524003 7181 ts_tablet_manager.cc:1397] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20260925 15:07:04.526238 7182 tablet_bootstrap.cc:654] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.527098 7182 log.cc:824] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:04.529265 7182 tablet_bootstrap.cc:492] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: No bootstrap required, opened a new log
I20260925 15:07:04.529503 7182 ts_tablet_manager.cc:1397] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.004s
I20260925 15:07:04.532492 7180 raft_consensus.cc:348] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.532905 7180 raft_consensus.cc:374] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.533021 7180 raft_consensus.cc:729] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987a49e415e455380387cf2beca770d, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.533550 7180 consensus_queue.cc:260] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.533975 7181 raft_consensus.cc:348] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.534613 7181 raft_consensus.cc:374] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.534772 7181 raft_consensus.cc:729] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.535661 7181 consensus_queue.cc:260] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.536768 7022 heartbeater.cc:503] Master 127.4.205.254:42379 was elected leader, sending a full tablet report...
I20260925 15:07:04.537631 7181 ts_tablet_manager.cc:1428] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260925 15:07:04.538687 7182 raft_consensus.cc:348] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.539052 7182 raft_consensus.cc:374] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.539063 7180 ts_tablet_manager.cc:1428] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260925 15:07:04.539139 7182 raft_consensus.cc:729] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40f0a8ef4614255b66b08055eec0113, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.539680 7182 consensus_queue.cc:260] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.540829 7161 heartbeater.cc:503] Master 127.4.205.254:42379 was elected leader, sending a full tablet report...
I20260925 15:07:04.541270 7182 ts_tablet_manager.cc:1428] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Time spent starting tablet: real 0.012s user 0.008s sys 0.001s
I20260925 15:07:04.545598 7187 raft_consensus.cc:482] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:04.545814 7187 raft_consensus.cc:504] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.547339 7187 leader_election.cc:288] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.554528 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.547947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:40127 (local address 127.4.205.195:37967)
0925 15:07:04.548099 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.548103 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.548394 (+ 291us) server_negotiation.cc:407] Connection header received
0925 15:07:04.548678 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.548683 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.548766 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.548850 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.549763 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.550646 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.552462 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.553273 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.553502 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.553606 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.554092 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.554258 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:07:04.554597 7189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.547615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:37967 (local address 127.4.205.194:40127)
0925 15:07:04.548285 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.548317 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.548506 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.549053 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.549059 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.549083 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.549493 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.549505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.550784 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.550790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.552313 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.552327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.553605 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.553613 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.553840 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.553958 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.554051 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.554173 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":61,"spinlock_wait_cycles":20352,"thread_start_us":486,"threads_started":1}
I20260925 15:07:04.554682 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.547604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:42003 (local address 127.4.205.193:46401)
0925 15:07:04.547829 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.547835 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.548660 (+ 825us) server_negotiation.cc:407] Connection header received
0925 15:07:04.549002 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.549006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.549073 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.549160 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.549975 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.550909 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.552584 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.553404 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.553605 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.553692 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.554300 (+ 608us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.554450 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:07:04.555127 7186 raft_consensus.cc:482] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:04.555359 7190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.547855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:46401 (local address 127.4.205.194:42003)
0925 15:07:04.548568 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.548585 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.548782 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.549206 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.549212 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.549255 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.549810 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.549825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.551041 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.551045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.552441 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.552455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.553852 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.553859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.554108 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.554151 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.554857 (+ 706us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.554980 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"mutex_wait_us":3,"thread_start_us":657,"threads_started":1}
I20260925 15:07:04.555408 7186 raft_consensus.cc:504] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.555840 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:04.556185 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d" is_pre_election: true
I20260925 15:07:04.556337 7114 raft_consensus.cc:2454] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:04.556613 6838 raft_consensus.cc:2454] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:04.557010 6903 leader_election.cc:302] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f40f0a8ef4614255b66b08055eec0113, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:04.557328 7186 leader_election.cc:288] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967), ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255)
I20260925 15:07:04.557691 7187 raft_consensus.cc:2781] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:04.557857 7187 raft_consensus.cc:482] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:04.557962 7187 raft_consensus.cc:3037] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.561041 7187 raft_consensus.cc:504] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.562114 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d"
I20260925 15:07:04.562098 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113"
I20260925 15:07:04.562296 7114 raft_consensus.cc:3037] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.562302 6838 raft_consensus.cc:3037] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.565510 7114 raft_consensus.cc:2454] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 1.
I20260925 15:07:04.565585 6838 raft_consensus.cc:2454] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 1.
I20260925 15:07:04.565968 6904 leader_election.cc:302] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:04.566732 7187 leader_election.cc:288] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Requested vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.566980 7187 raft_consensus.cc:2781] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:04.567137 7187 raft_consensus.cc:686] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 LEADER]: Becoming Leader. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Running, Role: LEADER
I20260925 15:07:04.567937 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.557921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:34929 (local address 127.4.205.194:35255)
0925 15:07:04.558113 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.558126 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.558807 (+ 681us) server_negotiation.cc:407] Connection header received
0925 15:07:04.559012 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.559022 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.559093 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.559188 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.559878 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.560950 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.564665 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.566381 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.566522 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.566584 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.567368 (+ 784us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.567564 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:04.567780 7187 consensus_queue.cc:237] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } }
I20260925 15:07:04.568902 7192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.557535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.4.205.193:34929)
0925 15:07:04.558665 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.558704 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.558873 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.559216 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.559223 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.559255 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.559722 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.559736 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.561088 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.561096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.564435 (+ 3339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.564455 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.566894 (+ 2439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.566902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.567100 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.567134 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.568403 (+ 1269us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.568558 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":11133,"threads_started":1}
I20260925 15:07:04.569825 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" is_pre_election: true
I20260925 15:07:04.575039 7177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.557874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:55313 (local address 127.4.205.195:37967)
0925 15:07:04.558040 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.558047 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.568724 (+ 10677us) server_negotiation.cc:407] Connection header received
0925 15:07:04.569085 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.569090 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.569164 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.569219 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:04.569944 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.571023 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.572575 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.574146 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.574318 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.574397 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.574672 (+ 275us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.574790 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:07:04.575138 7191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.557547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:37967 (local address 127.4.205.193:55313)
0925 15:07:04.568630 (+ 11083us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.568650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.568932 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.569402 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.569407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.569428 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:04.569768 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.569781 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.571176 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.571182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.572384 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.572393 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.574345 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.574352 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.574524 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.574545 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.574651 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.574767 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10980,"mutex_wait_us":289,"thread_start_us":488,"threads_started":1}
I20260925 15:07:04.575721 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2793ad9ef704967b290de1ebfd7b6e5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:04.576028 7114 raft_consensus.cc:2379] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e987a49e415e455380387cf2beca770d in current term 1: Already voted for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in this term.
I20260925 15:07:04.576639 6763 leader_election.cc:302] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e987a49e415e455380387cf2beca770d; no voters: f40f0a8ef4614255b66b08055eec0113, ff6ab613b7fc49f8ae78a7388a9a3cb4
I20260925 15:07:04.576957 7186 raft_consensus.cc:2726] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:07:04.576687 6675 catalog_manager.cc:5495] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } health_report { overall_health: HEALTHY } } }
Sep 25 15:07:04 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348822, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:04.624213 6771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.611833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54892 (local address 127.4.205.254:42379)
0925 15:07:04.611988 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.611993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.612489 (+ 496us) server_negotiation.cc:407] Connection header received
0925 15:07:04.612623 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.612627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.612682 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.612759 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:04.613980 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.614671 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.616001 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.616424 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.616514 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.619513 (+ 2999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:04.619518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:04.619521 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:04.619548 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:04.622020 (+ 2472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.622635 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.622638 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.622640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.622754 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:04.623055 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:04.623057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:04.623059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:04.623278 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:04.623384 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.623739 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.623939 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":53}
I20260925 15:07:04.624233 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.611855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42379 (local address 127.0.0.1:54892)
0925 15:07:04.612384 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.612399 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.612516 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.612814 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.612817 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.613503 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:04.613843 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.613852 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.614778 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.614781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.615855 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.615867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.616592 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.616596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.616713 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.617280 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:04.617295 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:04.619356 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:04.622131 (+ 2775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.622140 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.622145 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.622543 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.622842 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:04.622844 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:04.622847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:04.622971 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:04.623376 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:04.623382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:04.623514 (+ 132us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.623657 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.623843 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":312,"threads_started":1}
I20260925 15:07:04.629130 6675 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:54892:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:07:04.630082 6675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:04.636451 6952 tablet_service.cc:1467] Processing CreateTablet for tablet 3885994233854081b04ae23c06f3fd2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5868edfa9fba4b6e882ac4f1fb1230bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:04.636376 6813 tablet_service.cc:1467] Processing CreateTablet for tablet 3885994233854081b04ae23c06f3fd2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5868edfa9fba4b6e882ac4f1fb1230bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:04.637238 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3885994233854081b04ae23c06f3fd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.637259 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3885994233854081b04ae23c06f3fd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.641853 7091 tablet_service.cc:1467] Processing CreateTablet for tablet 3885994233854081b04ae23c06f3fd2b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5868edfa9fba4b6e882ac4f1fb1230bc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:04.642565 7181 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: Bootstrap starting.
I20260925 15:07:04.642792 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3885994233854081b04ae23c06f3fd2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.643092 7180 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: Bootstrap starting.
I20260925 15:07:04.643874 7181 tablet_bootstrap.cc:654] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.644673 7180 tablet_bootstrap.cc:654] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.646328 7181 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: No bootstrap required, opened a new log
I20260925 15:07:04.646466 7181 ts_tablet_manager.cc:1397] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:07:04.646817 7180 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: No bootstrap required, opened a new log
I20260925 15:07:04.646948 7180 ts_tablet_manager.cc:1397] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 15:07:04.647146 7181 raft_consensus.cc:348] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.647337 7181 raft_consensus.cc:374] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.647428 7181 raft_consensus.cc:729] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.647660 7181 consensus_queue.cc:260] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.647625 7180 raft_consensus.cc:348] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.648052 7187 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.648223 7181 ts_tablet_manager.cc:1428] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:07:04.648342 7187 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.647850 7180 raft_consensus.cc:374] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.648576 7180 raft_consensus.cc:729] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987a49e415e455380387cf2beca770d, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.648815 7180 consensus_queue.cc:260] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.649530 7180 ts_tablet_manager.cc:1428] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:04.649423 7186 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.649715 7186 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.650015 7182 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: Bootstrap starting.
I20260925 15:07:04.651598 7182 tablet_bootstrap.cc:654] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.653641 7182 tablet_bootstrap.cc:492] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: No bootstrap required, opened a new log
I20260925 15:07:04.653785 7182 ts_tablet_manager.cc:1397] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 15:07:04.654333 7182 raft_consensus.cc:348] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.654512 7182 raft_consensus.cc:374] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.654642 7182 raft_consensus.cc:729] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40f0a8ef4614255b66b08055eec0113, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.654883 7182 consensus_queue.cc:260] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.655474 7182 ts_tablet_manager.cc:1428] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:07:04.655311 7188 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.655660 7188 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: This TxnStatusTablet replica's current role is: FOLLOWER
W20260925 15:07:04.662462 7162 tablet.cc:2369] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:04.669592 7187 raft_consensus.cc:482] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:04.669770 7187 raft_consensus.cc:504] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.670269 7187 leader_election.cc:288] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.670722 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3885994233854081b04ae23c06f3fd2b" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:04.670719 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3885994233854081b04ae23c06f3fd2b" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d" is_pre_election: true
I20260925 15:07:04.670928 7114 raft_consensus.cc:2454] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:04.670964 6838 raft_consensus.cc:2454] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:04.671290 6903 leader_election.cc:302] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f40f0a8ef4614255b66b08055eec0113, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:04.671530 7187 raft_consensus.cc:2781] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:04.671623 7187 raft_consensus.cc:482] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:04.671723 7187 raft_consensus.cc:3037] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.673669 7187 raft_consensus.cc:504] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.674208 7187 leader_election.cc:288] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Requested vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.674580 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3885994233854081b04ae23c06f3fd2b" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d"
I20260925 15:07:04.674604 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3885994233854081b04ae23c06f3fd2b" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113"
I20260925 15:07:04.674777 6838 raft_consensus.cc:3037] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.674790 7114 raft_consensus.cc:3037] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.676954 7114 raft_consensus.cc:2454] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 1.
I20260925 15:07:04.677008 6838 raft_consensus.cc:2454] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 1.
I20260925 15:07:04.677313 6903 leader_election.cc:302] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f40f0a8ef4614255b66b08055eec0113, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:04.677618 7187 raft_consensus.cc:2781] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:04.677802 7187 raft_consensus.cc:686] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 LEADER]: Becoming Leader. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Running, Role: LEADER
I20260925 15:07:04.678045 7187 consensus_queue.cc:237] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.679653 7193 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: TxnStatusTablet state changed. Reason: New leader ff6ab613b7fc49f8ae78a7388a9a3cb4. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.680017 7193 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:04.680629 7205 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:07:04.680910 7205 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:07:04.681229 6674 catalog_manager.cc:5495] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
W20260925 15:07:04.682109 7023 tablet.cc:2369] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:04.692685 6884 tablet.cc:2369] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:04.700484 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.694777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.0.0.1:58640)
0925 15:07:04.694979 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.694998 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.695102 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.695584 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.695587 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.695596 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:04.695751 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.695758 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.697155 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.697158 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.698158 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.698169 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.698978 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.698981 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.699117 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.699161 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:04.700004 (+ 843us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:04.700017 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.700121 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.700279 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:07:04.700711 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.694852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58640 (local address 127.4.205.194:35255)
0925 15:07:04.695016 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.695021 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.695045 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:07:04.695280 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.695285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.695400 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.695560 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:04.695903 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.697046 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.698281 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.698844 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.698983 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.699310 (+ 327us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:04.699908 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:04.700066 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.700139 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.700245 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:07:04.708117 6838 raft_consensus.cc:1261] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:04.708206 7114 raft_consensus.cc:1261] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:04.709016 7193 consensus_queue.cc:1035] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:04.709633 7187 consensus_queue.cc:1035] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:04.719034 7186 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: TxnStatusTablet state changed. Reason: New leader ff6ab613b7fc49f8ae78a7388a9a3cb4. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.719295 7186 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.722716 7208 mvcc.cc:205] Tried to move back new op lower bound from 7333268785995968512 to 7333268785883443200. Current Snapshot: MvccSnapshot[applied={T|T < 7333268785995968512}]
I20260925 15:07:04.724641 7188 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: TxnStatusTablet state changed. Reason: New leader ff6ab613b7fc49f8ae78a7388a9a3cb4. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.724915 7188 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.726330 7186 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.726555 7186 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P e987a49e415e455380387cf2beca770d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.729390 7188 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.729609 7188 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P f40f0a8ef4614255b66b08055eec0113: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:04.737149 7193 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.737447 7193 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:04.736426 7187 tablet_replica.cc:439] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } } }
I20260925 15:07:04.737815 7187 tablet_replica.cc:442] T 3885994233854081b04ae23c06f3fd2b P ff6ab613b7fc49f8ae78a7388a9a3cb4: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:04.745708 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.741192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.4.205.194:35255)
0925 15:07:04.741379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.741387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.741405 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:07:04.741640 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.741646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.741716 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.741818 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:04.742162 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.742878 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.744051 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.744445 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.744595 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.744990 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:04.745084 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:04.745186 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.745323 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.745402 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:07:04.760771 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.755326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:58851 (local address 127.4.205.194:35255)
0925 15:07:04.755481 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:04.755485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:04.755946 (+ 461us) server_negotiation.cc:407] Connection header received
0925 15:07:04.756130 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:04.756136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:04.756195 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:04.756283 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:04.756780 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.757500 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.758974 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.759398 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.759552 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.759947 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:04.760060 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:04.760155 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:04.760285 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 15:07:04.760432 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":50}
I20260925 15:07:04.760870 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:04.755266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.4.205.194:58851)
0925 15:07:04.755856 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:04.755869 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:04.756013 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:04.756292 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:04.756296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:04.756305 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:04.756637 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.756644 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.757689 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.757696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.758831 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:04.758848 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:04.759573 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:04.759579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:04.759694 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:04.759738 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:04.760155 (+ 417us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:04.760166 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:07:04.760270 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 15:07:04.760475 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":50,"thread_start_us":340,"threads_started":1}
I20260925 15:07:04.768589 7114 raft_consensus.cc:1261] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:04.768664 6838 raft_consensus.cc:1261] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:04.769101 7187 consensus_queue.cc:1035] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:04.769312 7193 consensus_queue.cc:1035] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:04.774314 7209 mvcc.cc:205] Tried to move back new op lower bound from 7333268786247696384 to 7333268785435365376. Current Snapshot: MvccSnapshot[applied={T|T < 7333268786247696384}]
I20260925 15:07:04.827157 6675 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:04.827428 6675 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:04.830736 6675 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Sending DeleteTablet for 3 replicas of tablet a2793ad9ef704967b290de1ebfd7b6e5
I20260925 15:07:04.831735 7091 tablet_service.cc:1514] Processing DeleteTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:04 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260925 15:07:04.831696 6813 tablet_service.cc:1514] Processing DeleteTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:04 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:33802
I20260925 15:07:04.832041 6951 tablet_service.cc:1514] Processing DeleteTablet for tablet a2793ad9ef704967b290de1ebfd7b6e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:04 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58624
I20260925 15:07:04.833320 6675 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:04.833902 6675 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:04.841760 7090 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 (DEFAULT_TABLE table=test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.842278 7090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd5038307ad46c980f6c2b5b58de173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.844738 7227 tablet_replica.cc:330] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: stopping tablet replica
I20260925 15:07:04.845909 7227 raft_consensus.cc:2229] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:04.846450 6812 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 (DEFAULT_TABLE table=test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.846678 6952 tablet_service.cc:1467] Processing CreateTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 (DEFAULT_TABLE table=test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:04.847013 6812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd5038307ad46c980f6c2b5b58de173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.847040 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fd5038307ad46c980f6c2b5b58de173. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:04.847770 7227 raft_consensus.cc:2258] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:04.853080 7180 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Bootstrap starting.
I20260925 15:07:04.854768 7180 tablet_bootstrap.cc:654] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.857534 7227 ts_tablet_manager.cc:1905] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:04.861212 7182 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Bootstrap starting.
I20260925 15:07:04.861701 7181 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Bootstrap starting.
I20260925 15:07:04.862902 7182 tablet_bootstrap.cc:654] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.863462 7181 tablet_bootstrap.cc:654] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:04.867568 7180 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: No bootstrap required, opened a new log
I20260925 15:07:04.867767 7180 ts_tablet_manager.cc:1397] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.002s
I20260925 15:07:04.868726 7180 raft_consensus.cc:348] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.869051 7180 raft_consensus.cc:374] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.869118 7180 raft_consensus.cc:729] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987a49e415e455380387cf2beca770d, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.869422 7180 consensus_queue.cc:260] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.870217 7180 ts_tablet_manager.cc:1428] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:04.874487 7225 tablet_replica.cc:330] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: stopping tablet replica
I20260925 15:07:04.875059 7225 raft_consensus.cc:2229] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:04.875500 7225 raft_consensus.cc:2258] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:04.876327 7226 tablet_replica.cc:330] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: stopping tablet replica
I20260925 15:07:04.876760 7226 raft_consensus.cc:2229] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:04.877135 7226 raft_consensus.cc:2258] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:04.877579 7182 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: No bootstrap required, opened a new log
I20260925 15:07:04.877808 7182 ts_tablet_manager.cc:1397] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.008s
I20260925 15:07:04.878139 7225 ts_tablet_manager.cc:1905] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:04.878834 7182 raft_consensus.cc:348] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.879201 7182 raft_consensus.cc:374] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.879302 7182 raft_consensus.cc:729] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40f0a8ef4614255b66b08055eec0113, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.879388 7226 ts_tablet_manager.cc:1905] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:04.879664 7182 consensus_queue.cc:260] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.882202 7182 ts_tablet_manager.cc:1428] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:07:04.884104 7181 tablet_bootstrap.cc:492] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: No bootstrap required, opened a new log
I20260925 15:07:04.884352 7181 ts_tablet_manager.cc:1397] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.005s
I20260925 15:07:04.885164 7181 raft_consensus.cc:348] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.885425 7181 raft_consensus.cc:374] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:04.885512 7181 raft_consensus.cc:729] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Initialized, Role: FOLLOWER
I20260925 15:07:04.885905 7225 ts_tablet_manager.cc:1918] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:04.885802 7181 consensus_queue.cc:260] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.886152 7225 log.cc:1196] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/a2793ad9ef704967b290de1ebfd7b6e5
I20260925 15:07:04.888078 7225 ts_tablet_manager.cc:1939] T a2793ad9ef704967b290de1ebfd7b6e5 P e987a49e415e455380387cf2beca770d: Deleting consensus metadata
I20260925 15:07:04.888191 7181 ts_tablet_manager.cc:1428] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260925 15:07:04.888552 7227 ts_tablet_manager.cc:1918] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:04.888748 7227 log.cc:1196] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/a2793ad9ef704967b290de1ebfd7b6e5
I20260925 15:07:04.889292 7227 ts_tablet_manager.cc:1939] T a2793ad9ef704967b290de1ebfd7b6e5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting consensus metadata
I20260925 15:07:04.890197 6661 catalog_manager.cc:4841] TS e987a49e415e455380387cf2beca770d (127.4.205.193:46401): tablet a2793ad9ef704967b290de1ebfd7b6e5 (table test-table [id=2c7ca41558e349b680037eb28a775ed3]) successfully deleted
I20260925 15:07:04.891732 6662 catalog_manager.cc:4841] TS ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255): tablet a2793ad9ef704967b290de1ebfd7b6e5 (table test-table [id=2c7ca41558e349b680037eb28a775ed3]) successfully deleted
I20260925 15:07:04.894773 7226 ts_tablet_manager.cc:1918] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:04.894917 7226 log.cc:1196] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/a2793ad9ef704967b290de1ebfd7b6e5
I20260925 15:07:04.895320 7226 ts_tablet_manager.cc:1939] T a2793ad9ef704967b290de1ebfd7b6e5 P f40f0a8ef4614255b66b08055eec0113: Deleting consensus metadata
I20260925 15:07:04.896731 6660 catalog_manager.cc:4841] TS f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967): tablet a2793ad9ef704967b290de1ebfd7b6e5 (table test-table [id=2c7ca41558e349b680037eb28a775ed3]) successfully deleted
I20260925 15:07:04.994788 7186 raft_consensus.cc:482] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:04.995007 7186 raft_consensus.cc:504] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:04.995795 7186 leader_election.cc:288] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:04.996323 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd5038307ad46c980f6c2b5b58de173" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" is_pre_election: true
I20260925 15:07:04.996342 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd5038307ad46c980f6c2b5b58de173" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:04.996572 7114 raft_consensus.cc:2454] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 0.
I20260925 15:07:04.996626 6977 raft_consensus.cc:2454] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 0.
I20260925 15:07:04.997008 6763 leader_election.cc:302] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, f40f0a8ef4614255b66b08055eec0113; no voters:
I20260925 15:07:04.997305 7186 raft_consensus.cc:2781] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:04.997453 7186 raft_consensus.cc:482] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:04.997594 7186 raft_consensus.cc:3037] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:04.999943 7186 raft_consensus.cc:504] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:05.000627 7186 leader_election.cc:288] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 election: Requested vote from peers ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:05.001109 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd5038307ad46c980f6c2b5b58de173" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
I20260925 15:07:05.001123 7114 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fd5038307ad46c980f6c2b5b58de173" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113"
I20260925 15:07:05.001350 7114 raft_consensus.cc:3037] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:05.001375 6977 raft_consensus.cc:3037] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:05.003525 6977 raft_consensus.cc:2454] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 1.
I20260925 15:07:05.003925 6765 leader_election.cc:302] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:05.004217 7186 raft_consensus.cc:2781] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:05.004223 7114 raft_consensus.cc:2454] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 1.
I20260925 15:07:05.004786 7186 raft_consensus.cc:686] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 1 LEADER]: Becoming Leader. State: Replica: e987a49e415e455380387cf2beca770d, State: Running, Role: LEADER
I20260925 15:07:05.005174 7186 consensus_queue.cc:237] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:05.008502 6675 catalog_manager.cc:5495] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d reported cstate change: term changed from 0 to 1, leader changed from <none> to e987a49e415e455380387cf2beca770d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "e987a49e415e455380387cf2beca770d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:05.032765 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:05.027605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33816 (local address 127.4.205.193:46401)
0925 15:07:05.027766 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:05.027772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:05.027998 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:07:05.028232 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:05.028238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:05.028318 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:05.028464 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:05.028827 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.029570 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.030794 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.031171 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.031310 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:05.031704 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:05.032212 (+ 508us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:05.032364 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:05.032414 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 15:07:05.032485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:07:05.047324 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:05.042295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42937 (local address 127.4.205.194:35255)
0925 15:07:05.042512 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:05.042524 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:05.042791 (+ 267us) server_negotiation.cc:407] Connection header received
0925 15:07:05.042976 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:05.042983 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:05.043063 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:05.043157 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:05.043583 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.044278 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.045611 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.046046 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.046183 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:05.046578 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:05.046679 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:05.046775 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:05.046930 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:07:05.047010 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:07:05.047417 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:05.042197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.4.205.193:42937)
0925 15:07:05.042702 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:05.042716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:05.042855 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:05.043173 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:05.043177 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:05.043197 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:05.043450 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:05.043458 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.044399 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.044407 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:05.045456 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:05.045466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.046222 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.046231 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:05.046358 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:05.046402 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:05.046774 (+ 372us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:05.046785 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:07:05.046911 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:07:05.047028 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
I20260925 15:07:05.055948 7114 raft_consensus.cc:1261] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Refusing update from remote peer e987a49e415e455380387cf2beca770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:05.056341 6977 raft_consensus.cc:1261] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Refusing update from remote peer e987a49e415e455380387cf2beca770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:05.056984 7231 consensus_queue.cc:1035] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:05.057687 7186 consensus_queue.cc:1035] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:05.067176 7223 mvcc.cc:205] Tried to move back new op lower bound from 7333268787423391744 to 7333268787224993792. Current Snapshot: MvccSnapshot[applied={T|T < 7333268787423391744}]
I20260925 15:07:05.084932 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:05.079581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:46401 (local address 127.4.205.194:52485)
0925 15:07:05.079820 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:05.079835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:05.079927 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:05.080244 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:05.080247 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:05.080256 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:05.080402 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:05.080408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.081797 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.081803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:05.082830 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:05.082840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.083659 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.083665 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:05.083859 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:05.083910 (+ 51us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:05.084316 (+ 406us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:05.084333 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:07:05.084436 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:07:05.084595 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:07:05.084913 7175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:05.079509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:52485 (local address 127.4.205.193:46401)
0925 15:07:05.079666 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:05.079672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:05.079911 (+ 239us) server_negotiation.cc:407] Connection header received
0925 15:07:05.080040 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:05.080046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:05.080125 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:05.080228 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:05.080545 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.081667 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.082960 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:05.083522 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:05.083637 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:05.084081 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:05.084220 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:05.084314 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:05.084593 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:07:05.084657 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 15:07:05.086097 7209 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:07:06.675083 6676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:06.675323 6676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54886:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:06.679503 6676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Sending DeleteTablet for 3 replicas of tablet 6fd5038307ad46c980f6c2b5b58de173
I20260925 15:07:06.680162 6813 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:06 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:33802
I20260925 15:07:06.680202 6952 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:06 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58624
I20260925 15:07:06.680370 7091 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fd5038307ad46c980f6c2b5b58de173 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:06 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260925 15:07:06.691877 7262 tablet_replica.cc:330] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: stopping tablet replica
I20260925 15:07:06.692359 7262 raft_consensus.cc:2229] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:06.692731 7262 raft_consensus.cc:2258] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:06.694056 7262 ts_tablet_manager.cc:1905] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:06.698148 7263 tablet_replica.cc:330] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: stopping tablet replica
I20260925 15:07:06.698398 7261 tablet_replica.cc:330] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: stopping tablet replica
I20260925 15:07:06.698413 7263 raft_consensus.cc:2229] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:06.698592 7261 raft_consensus.cc:2229] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:06.699040 7261 raft_consensus.cc:2258] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:06.699079 7263 raft_consensus.cc:2258] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:06.699936 7261 ts_tablet_manager.cc:1905] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:06.700723 7263 ts_tablet_manager.cc:1905] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:07:06 dist-test-slave-64fx krb5kdc[6631](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20260925 15:07:06.731537 7261 ts_tablet_manager.cc:1918] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:06.731640 7261 log.cc:1196] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/6fd5038307ad46c980f6c2b5b58de173
I20260925 15:07:06.732057 7261 ts_tablet_manager.cc:1939] T 6fd5038307ad46c980f6c2b5b58de173 P e987a49e415e455380387cf2beca770d: Deleting consensus metadata
I20260925 15:07:06.740759 7262 ts_tablet_manager.cc:1918] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:06.740824 7263 ts_tablet_manager.cc:1918] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:06.740890 7262 log.cc:1196] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/6fd5038307ad46c980f6c2b5b58de173
I20260925 15:07:06.740957 7263 log.cc:1196] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/6fd5038307ad46c980f6c2b5b58de173
I20260925 15:07:06.741438 7263 ts_tablet_manager.cc:1939] T 6fd5038307ad46c980f6c2b5b58de173 P f40f0a8ef4614255b66b08055eec0113: Deleting consensus metadata
Sep 25 15:07:06 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:06.742039 7262 ts_tablet_manager.cc:1939] T 6fd5038307ad46c980f6c2b5b58de173 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting consensus metadata
I20260925 15:07:06.748003 7271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:06.733647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.4.205.254:42379)
0925 15:07:06.734253 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:06.734262 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:06.734305 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:07:06.734532 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:06.734537 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:06.734594 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:06.734706 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:06.736034 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.736776 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.738057 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.738464 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.738604 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:06.742499 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:06.742504 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:06.742507 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:06.742535 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:06.745440 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:06.746110 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:06.746117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:06.746123 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:06.746235 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:06.746651 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:06.746660 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:06.746662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:06.746921 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:06.747168 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:06.747450 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:07:06.747653 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260925 15:07:06.751055 6676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:06.751621 6676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:06.757498 6813 tablet_service.cc:1467] Processing CreateTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 (DEFAULT_TABLE table=test-table [id=e2af37fc04194924b0a344705575e79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:06.757539 6952 tablet_service.cc:1467] Processing CreateTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 (DEFAULT_TABLE table=test-table [id=e2af37fc04194924b0a344705575e79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:06.757871 7091 tablet_service.cc:1467] Processing CreateTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 (DEFAULT_TABLE table=test-table [id=e2af37fc04194924b0a344705575e79c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:06.757977 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01afa35cef4258bbba88b1b4a097a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:06.758002 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01afa35cef4258bbba88b1b4a097a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:06.758215 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad01afa35cef4258bbba88b1b4a097a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:06.764647 7272 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Bootstrap starting.
I20260925 15:07:06.766026 7274 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Bootstrap starting.
I20260925 15:07:06.766749 7273 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Bootstrap starting.
I20260925 15:07:06.767112 7272 tablet_bootstrap.cc:654] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:06.767629 7274 tablet_bootstrap.cc:654] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:06.768261 7273 tablet_bootstrap.cc:654] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:06.769409 7272 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: No bootstrap required, opened a new log
I20260925 15:07:06.769614 7272 ts_tablet_manager.cc:1397] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 15:07:06.770309 7272 raft_consensus.cc:348] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.770556 7272 raft_consensus.cc:374] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:06.770646 7272 raft_consensus.cc:729] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987a49e415e455380387cf2beca770d, State: Initialized, Role: FOLLOWER
I20260925 15:07:06.770767 7274 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: No bootstrap required, opened a new log
I20260925 15:07:06.770905 7274 ts_tablet_manager.cc:1397] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:07:06.770869 7272 consensus_queue.cc:260] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.771044 7273 tablet_bootstrap.cc:492] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: No bootstrap required, opened a new log
I20260925 15:07:06.771334 7273 ts_tablet_manager.cc:1397] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:07:06.771442 7272 ts_tablet_manager.cc:1428] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:07:06.771646 7274 raft_consensus.cc:348] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.771869 7274 raft_consensus.cc:374] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:06.771960 7274 raft_consensus.cc:729] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40f0a8ef4614255b66b08055eec0113, State: Initialized, Role: FOLLOWER
I20260925 15:07:06.772150 7273 raft_consensus.cc:348] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.772172 7274 consensus_queue.cc:260] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.772374 7273 raft_consensus.cc:374] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:06.772475 7273 raft_consensus.cc:729] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Initialized, Role: FOLLOWER
I20260925 15:07:06.772780 7273 consensus_queue.cc:260] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.773054 7274 ts_tablet_manager.cc:1428] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:06.774111 7273 ts_tablet_manager.cc:1428] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:06.808919 6661 catalog_manager.cc:4841] TS e987a49e415e455380387cf2beca770d (127.4.205.193:46401): tablet 6fd5038307ad46c980f6c2b5b58de173 (table test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]) successfully deleted
I20260925 15:07:06.808919 6660 catalog_manager.cc:4841] TS f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967): tablet 6fd5038307ad46c980f6c2b5b58de173 (table test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]) successfully deleted
I20260925 15:07:06.808947 6662 catalog_manager.cc:4841] TS ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255): tablet 6fd5038307ad46c980f6c2b5b58de173 (table test-table [id=2f0a2cd74c3c46a7911b2bdfcc46aefe]) successfully deleted
I20260925 15:07:06.899986 7278 raft_consensus.cc:482] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:06.900257 7278 raft_consensus.cc:504] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.901983 7278 leader_election.cc:288] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255)
I20260925 15:07:06.906353 7256 raft_consensus.cc:482] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:06.906611 7256 raft_consensus.cc:504] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.907496 7256 leader_election.cc:288] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:06.915400 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d" is_pre_election: true
I20260925 15:07:06.915652 6838 raft_consensus.cc:2454] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:06.916165 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:06.916304 6904 leader_election.cc:302] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters:
I20260925 15:07:06.916463 7113 raft_consensus.cc:2454] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 0.
I20260925 15:07:06.916630 7256 raft_consensus.cc:2781] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:06.916755 7256 raft_consensus.cc:482] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:06.916855 7256 raft_consensus.cc:3037] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:06.924642 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:06.902397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:54421 (local address 127.4.205.194:35255)
0925 15:07:06.914724 (+ 12327us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:06.914733 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:06.915964 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 15:07:06.916049 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:06.916053 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:06.916129 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:06.916220 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:06.917598 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.918901 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.922091 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.923339 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.923501 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:06.923577 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:06.924182 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:07:06.924318 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12149,"thread_start_us":12361,"threads_started":1}
I20260925 15:07:06.924638 7281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:06.902318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:50139 (local address 127.4.205.193:46401)
0925 15:07:06.916981 (+ 14663us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:06.916987 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:06.917013 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:07:06.917078 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:06.917085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:06.917148 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:06.917886 (+ 738us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:06.918054 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.919097 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.922091 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.923378 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.923531 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:06.923601 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:06.924192 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:07:06.924324 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14540,"thread_start_us":14426,"threads_started":1}
I20260925 15:07:06.924744 7279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:06.902202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:46401 (local address 127.4.205.195:50139)
0925 15:07:06.915755 (+ 13553us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:06.915776 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:06.915910 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:06.917302 (+ 1392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:06.917310 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:06.917337 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:06.917727 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:06.917740 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.919205 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.919216 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:06.921946 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:06.921960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.923601 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.923606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:06.923955 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:06.924049 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 15:07:06.924137 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:07:06.924310 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13379,"mutex_wait_us":174,"thread_start_us":12843,"threads_started":1}
I20260925 15:07:06.924778 7280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:06.902198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35255 (local address 127.4.205.195:54421)
0925 15:07:06.915340 (+ 13142us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:06.915368 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:06.915519 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:06.916492 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:06.916497 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:06.916520 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:06.917373 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:06.917390 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.919044 (+ 1654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.919052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:06.921946 (+ 2894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:06.921960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:06.923594 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:06.923600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:06.923960 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:06.924053 (+ 93us) client_negotiation.cc:769] Sending connection context
0925 15:07:06.924143 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:07:06.924310 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12999,"mutex_wait_us":266,"thread_start_us":13420,"threads_started":1}
I20260925 15:07:06.925282 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "f40f0a8ef4614255b66b08055eec0113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d" is_pre_election: true
I20260925 15:07:06.925482 6838 raft_consensus.cc:2454] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f40f0a8ef4614255b66b08055eec0113 in term 0.
I20260925 15:07:06.925551 6977 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "f40f0a8ef4614255b66b08055eec0113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" is_pre_election: true
I20260925 15:07:06.925954 7043 leader_election.cc:302] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, f40f0a8ef4614255b66b08055eec0113; no voters:
I20260925 15:07:06.926261 7278 raft_consensus.cc:2781] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:06.926385 7278 raft_consensus.cc:482] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:06.926476 7278 raft_consensus.cc:3037] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:06.930789 7278 raft_consensus.cc:504] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.930751 7256 raft_consensus.cc:504] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:06.931466 7256 leader_election.cc:288] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Requested vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:06.931533 7278 leader_election.cc:288] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [CANDIDATE]: Term 1 election: Requested vote from peers e987a49e415e455380387cf2beca770d (127.4.205.193:46401), ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255)
I20260925 15:07:06.932178 6977 raft_consensus.cc:2379] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f40f0a8ef4614255b66b08055eec0113 in current term 1: Already voted for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in this term.
I20260925 15:07:06.932262 6838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d"
I20260925 15:07:06.932489 6838 raft_consensus.cc:3037] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:06.932704 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113"
I20260925 15:07:06.932943 7113 raft_consensus.cc:2379] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in current term 1: Already voted for candidate f40f0a8ef4614255b66b08055eec0113 in this term.
I20260925 15:07:06.932896 6976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "f40f0a8ef4614255b66b08055eec0113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
I20260925 15:07:06.933151 6976 raft_consensus.cc:2379] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f40f0a8ef4614255b66b08055eec0113 in current term 1: Already voted for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in this term.
I20260925 15:07:06.933830 6837 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad01afa35cef4258bbba88b1b4a097a8" candidate_uuid: "f40f0a8ef4614255b66b08055eec0113" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e987a49e415e455380387cf2beca770d"
I20260925 15:07:06.934412 7043 leader_election.cc:302] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f40f0a8ef4614255b66b08055eec0113; no voters: e987a49e415e455380387cf2beca770d, ff6ab613b7fc49f8ae78a7388a9a3cb4
I20260925 15:07:06.935014 7278 raft_consensus.cc:2726] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 15:07:07.030076 6838 raft_consensus.cc:2454] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6ab613b7fc49f8ae78a7388a9a3cb4 in term 1.
I20260925 15:07:07.030777 6904 leader_election.cc:302] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e987a49e415e455380387cf2beca770d, ff6ab613b7fc49f8ae78a7388a9a3cb4; no voters: f40f0a8ef4614255b66b08055eec0113
I20260925 15:07:07.031174 7256 raft_consensus.cc:2781] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:07.031349 7256 raft_consensus.cc:686] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 LEADER]: Becoming Leader. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Running, Role: LEADER
I20260925 15:07:07.031741 7256 consensus_queue.cc:237] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.034762 6674 catalog_manager.cc:5495] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:07.065469 7282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:07.060906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58666 (local address 127.4.205.194:35255)
0925 15:07:07.061104 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:07.061111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:07.061130 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:07:07.061302 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:07.061308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:07.061395 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:07.061487 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:07.061876 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:07.062574 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:07.063844 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:07.064270 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:07.064413 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:07.064783 (+ 370us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:07.064896 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:07.064993 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:07.065133 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:07:07.065205 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:07:07.074208 6838 raft_consensus.cc:1261] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:07.074221 7113 raft_consensus.cc:1261] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Refusing update from remote peer ff6ab613b7fc49f8ae78a7388a9a3cb4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:07.074709 7242 consensus_queue.cc:1035] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:07.075039 7239 consensus_queue.cc:1035] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:07.115942 6676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:07.116184 6676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:07.118726 6676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Sending DeleteTablet for 3 replicas of tablet ad01afa35cef4258bbba88b1b4a097a8
I20260925 15:07:07.119470 6813 tablet_service.cc:1514] Processing DeleteTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:07 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:33802
I20260925 15:07:07.119618 7091 tablet_service.cc:1514] Processing DeleteTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:07 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260925 15:07:07.119617 6952 tablet_service.cc:1514] Processing DeleteTablet for tablet ad01afa35cef4258bbba88b1b4a097a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:07 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58624
I20260925 15:07:07.119752 7261 tablet_replica.cc:330] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: stopping tablet replica
I20260925 15:07:07.119879 7263 tablet_replica.cc:330] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: stopping tablet replica
I20260925 15:07:07.119879 7262 tablet_replica.cc:330] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: stopping tablet replica
I20260925 15:07:07.119967 7261 raft_consensus.cc:2229] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:07.120050 7263 raft_consensus.cc:2229] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:07.120054 7262 raft_consensus.cc:2229] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:07.120153 7261 raft_consensus.cc:2258] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:07.120210 7263 raft_consensus.cc:2258] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:07.120343 7262 raft_consensus.cc:2258] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:07.121222 7263 ts_tablet_manager.cc:1905] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:07.121702 7262 ts_tablet_manager.cc:1905] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:07.121634 6676 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:07.122144 6676 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:07.123672 7261 ts_tablet_manager.cc:1905] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:07.129709 6813 tablet_service.cc:1467] Processing CreateTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 (DEFAULT_TABLE table=test-table [id=8eede9fad05b427f942d12659eec518d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:07.129885 6952 tablet_service.cc:1467] Processing CreateTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 (DEFAULT_TABLE table=test-table [id=8eede9fad05b427f942d12659eec518d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:07.130234 7091 tablet_service.cc:1467] Processing CreateTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 (DEFAULT_TABLE table=test-table [id=8eede9fad05b427f942d12659eec518d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:07.132344 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f55e3d5bff74fab9709b6f53652e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:07.132371 6813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f55e3d5bff74fab9709b6f53652e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:07.137679 6952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f55e3d5bff74fab9709b6f53652e2f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:07.141502 7274 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Bootstrap starting.
I20260925 15:07:07.142259 7272 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Bootstrap starting.
I20260925 15:07:07.142814 7274 tablet_bootstrap.cc:654] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:07.143898 7272 tablet_bootstrap.cc:654] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:07.145080 7274 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: No bootstrap required, opened a new log
I20260925 15:07:07.145228 7274 ts_tablet_manager.cc:1397] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:07:07.145867 7272 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: No bootstrap required, opened a new log
I20260925 15:07:07.145859 7274 raft_consensus.cc:348] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.146027 7272 ts_tablet_manager.cc:1397] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260925 15:07:07.146111 7274 raft_consensus.cc:374] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:07.146211 7274 raft_consensus.cc:729] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f40f0a8ef4614255b66b08055eec0113, State: Initialized, Role: FOLLOWER
I20260925 15:07:07.146471 7274 consensus_queue.cc:260] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.146916 7274 ts_tablet_manager.cc:1428] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:07.146663 7272 raft_consensus.cc:348] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.147329 7272 raft_consensus.cc:374] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:07.147425 7272 raft_consensus.cc:729] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e987a49e415e455380387cf2beca770d, State: Initialized, Role: FOLLOWER
I20260925 15:07:07.147640 7272 consensus_queue.cc:260] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.148500 7272 ts_tablet_manager.cc:1428] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:07.236076 7262 ts_tablet_manager.cc:1918] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:07.236076 7263 ts_tablet_manager.cc:1918] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:07.236076 7261 ts_tablet_manager.cc:1918] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:07.236253 7261 log.cc:1196] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/ad01afa35cef4258bbba88b1b4a097a8
I20260925 15:07:07.236258 7263 log.cc:1196] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/ad01afa35cef4258bbba88b1b4a097a8
I20260925 15:07:07.236258 7262 log.cc:1196] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/ad01afa35cef4258bbba88b1b4a097a8
I20260925 15:07:07.236688 7261 ts_tablet_manager.cc:1939] T ad01afa35cef4258bbba88b1b4a097a8 P e987a49e415e455380387cf2beca770d: Deleting consensus metadata
I20260925 15:07:07.236699 7262 ts_tablet_manager.cc:1939] T ad01afa35cef4258bbba88b1b4a097a8 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting consensus metadata
I20260925 15:07:07.236719 7263 ts_tablet_manager.cc:1939] T ad01afa35cef4258bbba88b1b4a097a8 P f40f0a8ef4614255b66b08055eec0113: Deleting consensus metadata
I20260925 15:07:07.237826 6661 catalog_manager.cc:4841] TS e987a49e415e455380387cf2beca770d (127.4.205.193:46401): tablet ad01afa35cef4258bbba88b1b4a097a8 (table test-table [id=e2af37fc04194924b0a344705575e79c]) successfully deleted
I20260925 15:07:07.237797 6660 catalog_manager.cc:4841] TS f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967): tablet ad01afa35cef4258bbba88b1b4a097a8 (table test-table [id=e2af37fc04194924b0a344705575e79c]) successfully deleted
I20260925 15:07:07.238175 6662 catalog_manager.cc:4841] TS ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255): tablet ad01afa35cef4258bbba88b1b4a097a8 (table test-table [id=e2af37fc04194924b0a344705575e79c]) successfully deleted
I20260925 15:07:07.240047 7273 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Bootstrap starting.
I20260925 15:07:07.241271 7273 tablet_bootstrap.cc:654] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:07.242934 7273 tablet_bootstrap.cc:492] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: No bootstrap required, opened a new log
I20260925 15:07:07.243098 7273 ts_tablet_manager.cc:1397] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:07:07.244031 7273 raft_consensus.cc:348] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.244232 7273 raft_consensus.cc:374] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:07.244310 7273 raft_consensus.cc:729] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6ab613b7fc49f8ae78a7388a9a3cb4, State: Initialized, Role: FOLLOWER
I20260925 15:07:07.244554 7273 consensus_queue.cc:260] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.244990 7273 ts_tablet_manager.cc:1428] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:07:07.371526 7251 raft_consensus.cc:482] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:07.371766 7251 raft_consensus.cc:504] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.372745 7251 leader_election.cc:288] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:07.373193 6976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f55e3d5bff74fab9709b6f53652e2f5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" is_pre_election: true
I20260925 15:07:07.373180 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f55e3d5bff74fab9709b6f53652e2f5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113" is_pre_election: true
I20260925 15:07:07.373413 6976 raft_consensus.cc:2454] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 0.
I20260925 15:07:07.373457 7113 raft_consensus.cc:2454] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 0.
I20260925 15:07:07.373966 6763 leader_election.cc:302] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, f40f0a8ef4614255b66b08055eec0113; no voters:
I20260925 15:07:07.374230 7251 raft_consensus.cc:2781] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:07.374346 7251 raft_consensus.cc:482] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:07.374421 7251 raft_consensus.cc:3037] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:07.376410 7251 raft_consensus.cc:504] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.377112 7251 leader_election.cc:288] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 election: Requested vote from peers ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255), f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967)
I20260925 15:07:07.377391 7113 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f55e3d5bff74fab9709b6f53652e2f5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f40f0a8ef4614255b66b08055eec0113"
I20260925 15:07:07.377393 6976 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f55e3d5bff74fab9709b6f53652e2f5" candidate_uuid: "e987a49e415e455380387cf2beca770d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4"
I20260925 15:07:07.377578 6976 raft_consensus.cc:3037] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:07.377671 7113 raft_consensus.cc:3037] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:07.380384 7113 raft_consensus.cc:2454] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 1.
I20260925 15:07:07.380384 6976 raft_consensus.cc:2454] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e987a49e415e455380387cf2beca770d in term 1.
I20260925 15:07:07.380734 6763 leader_election.cc:302] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e987a49e415e455380387cf2beca770d, f40f0a8ef4614255b66b08055eec0113; no voters:
I20260925 15:07:07.381014 7251 raft_consensus.cc:2781] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:07.381486 7251 raft_consensus.cc:686] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 1 LEADER]: Becoming Leader. State: Replica: e987a49e415e455380387cf2beca770d, State: Running, Role: LEADER
I20260925 15:07:07.381783 7251 consensus_queue.cc:237] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } }
I20260925 15:07:07.384513 6674 catalog_manager.cc:5495] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d reported cstate change: term changed from 0 to 1, leader changed from <none> to e987a49e415e455380387cf2beca770d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "e987a49e415e455380387cf2beca770d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e987a49e415e455380387cf2beca770d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 46401 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:07.452934 7294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:07.445575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33830 (local address 127.4.205.193:46401)
0925 15:07:07.446809 (+ 1234us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:07.446815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:07.446830 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:07.446882 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:07.446887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:07.446951 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:07.447037 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:07.447560 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:07.448592 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:07.450852 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:07.451500 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:07.451610 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:07.452179 (+ 569us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:07.452294 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:07.452394 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:07.452570 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:07:07.452638 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1133,"thread_start_us":1047,"threads_started":1}
I20260925 15:07:07.463327 7113 raft_consensus.cc:1261] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Refusing update from remote peer e987a49e415e455380387cf2beca770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:07.463464 6976 raft_consensus.cc:1261] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Refusing update from remote peer e987a49e415e455380387cf2beca770d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:07.463781 7251 consensus_queue.cc:1035] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f40f0a8ef4614255b66b08055eec0113" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:07.464303 7251 consensus_queue.cc:1035] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff6ab613b7fc49f8ae78a7388a9a3cb4" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:07:07.502126 7244 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 15:07:09.081001 6674 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:09.081243 6674 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54902:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:09.085327 6674 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e0b47f88a848e19a0c45c585b83ba6: Sending DeleteTablet for 3 replicas of tablet 7f55e3d5bff74fab9709b6f53652e2f5
I20260925 15:07:09.086076 6813 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:09 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:33802
I20260925 15:07:09.086162 6951 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:09 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58624
I20260925 15:07:09.086483 7091 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f55e3d5bff74fab9709b6f53652e2f5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:09 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39978
I20260925 15:07:09.087105 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6747
I20260925 15:07:09.088332 7321 tablet_replica.cc:330] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: stopping tablet replica
I20260925 15:07:09.088523 7321 raft_consensus.cc:2229] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:09.088764 7321 raft_consensus.cc:2258] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:09.090505 7321 ts_tablet_manager.cc:1905] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:09.097503 7320 tablet_replica.cc:330] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: stopping tablet replica
I20260925 15:07:09.097728 7320 raft_consensus.cc:2229] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:09.098039 7320 raft_consensus.cc:2258] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:09.098205 7322 tablet_replica.cc:330] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: stopping tablet replica
I20260925 15:07:09.098398 7322 raft_consensus.cc:2229] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:09.098644 7322 raft_consensus.cc:2258] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:09.098882 7320 ts_tablet_manager.cc:1905] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:09.099650 7322 ts_tablet_manager.cc:1905] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:09.099828 7321 ts_tablet_manager.cc:1918] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:09.099969 7321 log.cc:1196] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/7f55e3d5bff74fab9709b6f53652e2f5
I20260925 15:07:09.100535 7321 ts_tablet_manager.cc:1939] T 7f55e3d5bff74fab9709b6f53652e2f5 P ff6ab613b7fc49f8ae78a7388a9a3cb4: Deleting consensus metadata
I20260925 15:07:09.101702 6662 catalog_manager.cc:4841] TS ff6ab613b7fc49f8ae78a7388a9a3cb4 (127.4.205.194:35255): tablet 7f55e3d5bff74fab9709b6f53652e2f5 (table test-table [id=8eede9fad05b427f942d12659eec518d]) successfully deleted
I20260925 15:07:09.106648 7322 ts_tablet_manager.cc:1918] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:09.106782 7322 log.cc:1196] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/7f55e3d5bff74fab9709b6f53652e2f5
I20260925 15:07:09.106817 7320 ts_tablet_manager.cc:1918] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 15:07:09.106954 7320 log.cc:1196] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/7f55e3d5bff74fab9709b6f53652e2f5
I20260925 15:07:09.107246 7322 ts_tablet_manager.cc:1939] T 7f55e3d5bff74fab9709b6f53652e2f5 P f40f0a8ef4614255b66b08055eec0113: Deleting consensus metadata
I20260925 15:07:09.107379 7320 ts_tablet_manager.cc:1939] T 7f55e3d5bff74fab9709b6f53652e2f5 P e987a49e415e455380387cf2beca770d: Deleting consensus metadata
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:09.108239 6660 catalog_manager.cc:4841] TS f40f0a8ef4614255b66b08055eec0113 (127.4.205.195:37967): tablet 7f55e3d5bff74fab9709b6f53652e2f5 (table test-table [id=8eede9fad05b427f942d12659eec518d]) successfully deleted
I20260925 15:07:09.108310 6661 catalog_manager.cc:4841] TS e987a49e415e455380387cf2beca770d (127.4.205.193:46401): tablet 7f55e3d5bff74fab9709b6f53652e2f5 (table test-table [id=8eede9fad05b427f942d12659eec518d]) successfully deleted
I20260925 15:07:09.114359 7324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:09.088532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33842 (local address 127.4.205.193:46401)
0925 15:07:09.097335 (+ 8803us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:09.097341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:09.097366 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:09.097426 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:09.097431 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:09.097512 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:09.097650 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:09.101558 (+ 3908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.102597 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.103951 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.104420 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.104549 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:09.108906 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:09.108931 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:09.108947 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:09.108974 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:09.111720 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.112358 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.112370 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.112378 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.112506 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.112875 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.112887 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.112889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.113194 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:09.113479 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:09.113797 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:07:09.114009 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8661,"thread_start_us":8548,"threads_started":1}
W20260925 15:07:09.115003 6878 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33842
W20260925 15:07:09.115514 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6747: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:09.115698 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6747
I20260925 15:07:09.137943 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6887
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:09.152680 7325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:09.138613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.4.205.194:35255)
0925 15:07:09.139010 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:09.139014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:09.139027 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:07:09.139071 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:09.139074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:09.139130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:09.139208 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:09.140245 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.140978 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.142602 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.143200 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.143299 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:09.147633 (+ 4334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:09.147645 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:09.147658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:09.147685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:09.150431 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.151008 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.151015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.151017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.151136 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.151479 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.151482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.151484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.151787 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:09.151986 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:09.152188 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:07:09.152327 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":204,"threads_started":1}
W20260925 15:07:09.153288 7017 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58682
W20260925 15:07:09.153844 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6887: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:09.153957 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6887
I20260925 15:07:09.178252 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7026
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[6631](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348826, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:09.191869 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:09.178846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.4.205.195:37967)
0925 15:07:09.179273 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:09.179278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:09.179294 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:09.179344 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:09.179347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:09.179450 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:09.179530 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:09.180629 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.181372 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.182557 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:09.182944 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:09.183035 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:09.187084 (+ 4049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:09.187098 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:09.187113 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:09.187138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:09.189631 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.190151 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.190158 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.190160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.190302 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:09.190639 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:09.190642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:09.190644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:09.190986 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:09.191180 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:09.191394 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:07:09.191575 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":205,"threads_started":1}
W20260925 15:07:09.192461 7156 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39982
W20260925 15:07:09.193017 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7026: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:09.193158 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7026
I20260925 15:07:09.209946 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 6644
I20260925 15:07:09.210970 6716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:09.322921 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 6644
2026-09-25T15:07:09Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8083 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[7330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[7330](info): set up 2 sockets
Sep 25 15:07:09 dist-test-slave-64fx krb5kdc[7330](info): commencing operation
krb5kdc: starting...
W20260925 15:07:10.369782 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.010s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:10 dist-test-slave-64fx krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:10.540845 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37347
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:37347
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:10.843588 7343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:10.843844 7343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:10.843925 7343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:10.850805 7343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:10.850893 7343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:10.850936 7343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:10.851009 7343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:10.851068 7343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:10.860224 7343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:37347
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:10.861402 7343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:10.862700 7343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:10.873643 7348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:10.873714 7351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:10.873730 7349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:10.875100 7343 server_base.cc:1034] running on GCE node
I20260925 15:07:10.875723 7343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:10.876899 7343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:10.878073 7343 hybrid_clock.cc:648] HybridClock initialized: now 1790348830878021 us; error 47 us; skew 500 ppm
I20260925 15:07:10.878393 7343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:10 dist-test-slave-64fx krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348830, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:10.881193 7343 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:10.882630 7343 webserver.cc:460] Webserver started at http://127.4.205.254:46237/ using document root <none> and password file <none>
I20260925 15:07:10.883297 7343 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:10.883414 7343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:10.883704 7343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:10.886535 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "3cc5e7173f3a45228a62df494a30e9b6"
format_stamp: "Formatted at 2026-09-25 15:07:10 on dist-test-slave-64fx"
server_key: "42c644344b978e1e2108464c13bada07"
server_key_iv: "6b80fa9983d2222f6cb222a60c1a2c0a"
server_key_version: "encryptionkey@0"
I20260925 15:07:10.887224 7343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "3cc5e7173f3a45228a62df494a30e9b6"
format_stamp: "Formatted at 2026-09-25 15:07:10 on dist-test-slave-64fx"
server_key: "42c644344b978e1e2108464c13bada07"
server_key_iv: "6b80fa9983d2222f6cb222a60c1a2c0a"
server_key_version: "encryptionkey@0"
I20260925 15:07:10.891845 7343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:07:10.894816 7358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:10.895936 7343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:10.896104 7343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "3cc5e7173f3a45228a62df494a30e9b6"
format_stamp: "Formatted at 2026-09-25 15:07:10 on dist-test-slave-64fx"
server_key: "42c644344b978e1e2108464c13bada07"
server_key_iv: "6b80fa9983d2222f6cb222a60c1a2c0a"
server_key_version: "encryptionkey@0"
I20260925 15:07:10.896262 7343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:10.907514 7343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:10.910953 7343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:10.911202 7343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:10.927937 7343 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:37347
I20260925 15:07:10.927954 7409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:37347 every 8 connection(s)
I20260925 15:07:10.929157 7343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:10.932377 7410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:10.934290 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7343
I20260925 15:07:10.934502 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:10.935076 4919 external_mini_cluster.cc:1428] Setting key 68ec6e1e61bda4340b226c663990f02d
I20260925 15:07:10.943215 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6: Bootstrap starting.
I20260925 15:07:10.946741 7410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:10.947975 7410 log.cc:824] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:10 dist-test-slave-64fx krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:10.950575 7410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6: No bootstrap required, opened a new log
I20260925 15:07:10.956638 7413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:10.936963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.4.205.254:37347)
0925 15:07:10.937706 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:10.937716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:10.937753 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:07:10.939246 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:10.939265 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:10.939610 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:10.939928 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:10.941663 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:10.942851 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:10.944561 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:10.945214 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:10.945332 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:10.949997 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:10.950003 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:10.950015 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:10.950043 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:10.952851 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:10.953495 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:10.953517 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:10.953522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:10.953741 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:10.954113 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:10.954119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:10.954123 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:10.954608 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:10.954888 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:10.955331 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 15:07:10.955715 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":328,"threads_started":1}
I20260925 15:07:10.958034 7410 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } }
I20260925 15:07:10.958343 7410 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:10.958441 7410 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cc5e7173f3a45228a62df494a30e9b6, State: Initialized, Role: FOLLOWER
I20260925 15:07:10.958998 7410 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } }
I20260925 15:07:10.959204 7410 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:10.959360 7410 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:10.959527 7410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:10.961419 7410 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } }
I20260925 15:07:10.961902 7410 leader_election.cc:302] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3cc5e7173f3a45228a62df494a30e9b6; no voters:
I20260925 15:07:10.962464 7410 leader_election.cc:288] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:10.962613 7415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:10.963313 7415 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [term 1 LEADER]: Becoming Leader. State: Replica: 3cc5e7173f3a45228a62df494a30e9b6, State: Running, Role: LEADER
I20260925 15:07:10.963851 7415 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } }
I20260925 15:07:10.964322 7410 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:10.971004 7416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3cc5e7173f3a45228a62df494a30e9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } } }
I20260925 15:07:10.971300 7416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:10.971740 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3cc5e7173f3a45228a62df494a30e9b6. Latest consensus state: current_term: 1 leader_uuid: "3cc5e7173f3a45228a62df494a30e9b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3cc5e7173f3a45228a62df494a30e9b6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37347 } } }
I20260925 15:07:10.972023 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:10.972466 7424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:10.977360 7424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:10.984755 7424 catalog_manager.cc:1347] Generated new cluster ID: 1dfe8d77c6284afea2cc8f53464c814c
I20260925 15:07:10.984889 7424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:10.993880 7424 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:10.995064 7424 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:11.004591 7424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3cc5e7173f3a45228a62df494a30e9b6: Generated new TSK 0
I20260925 15:07:11.005446 7424 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:11.054854 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:11.361469 7436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:11.361763 7436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:11.361853 7436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:11.368829 7436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:11.368925 7436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:11.369023 7436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:11.378585 7436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:11.379786 7436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:11.381011 7436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:11.393457 7442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.393608 7444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.393608 7441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:11.394474 7436 server_base.cc:1034] running on GCE node
I20260925 15:07:11.394985 7436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:11.395771 7436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:11.396957 7436 hybrid_clock.cc:648] HybridClock initialized: now 1790348831396908 us; error 50 us; skew 500 ppm
I20260925 15:07:11.397233 7436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:11 dist-test-slave-64fx krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:11.400211 7436 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:11.401770 7436 webserver.cc:460] Webserver started at http://127.4.205.193:35859/ using document root <none> and password file <none>
I20260925 15:07:11.402392 7436 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:11.402514 7436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:11.402803 7436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:11.405454 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "83c125bb1b844fbf8bef37e3e7215e76"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "0d1628d639f1e051b0170cf95489057e"
server_key_iv: "aad8d59a0d641056d05ee89da5ecd9c8"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.406383 7436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "83c125bb1b844fbf8bef37e3e7215e76"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "0d1628d639f1e051b0170cf95489057e"
server_key_iv: "aad8d59a0d641056d05ee89da5ecd9c8"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.411271 7436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:07:11.414613 7451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.415925 7436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:11.416042 7436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "83c125bb1b844fbf8bef37e3e7215e76"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "0d1628d639f1e051b0170cf95489057e"
server_key_iv: "aad8d59a0d641056d05ee89da5ecd9c8"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.416200 7436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:11.431746 7436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:11.435423 7436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:11.435649 7436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:11.436494 7436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:11.437794 7436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:11.437872 7436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.437961 7436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:11.438004 7436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.469663 7436 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:41551
I20260925 15:07:11.469728 7563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:41551 every 8 connection(s)
I20260925 15:07:11.471148 7436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:11.480201 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7436
I20260925 15:07:11.480422 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:11.481137 4919 external_mini_cluster.cc:1428] Setting key 273c02fc13dbca7b9a3d26d37ea32f54
Sep 25 15:07:11 dist-test-slave-64fx krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:11.490446 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.473644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57313 (local address 127.4.205.254:37347)
0925 15:07:11.474086 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:11.474090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:11.475008 (+ 918us) server_negotiation.cc:407] Connection header received
0925 15:07:11.476733 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:11.476738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:11.476822 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:11.476931 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:11.478355 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.479013 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.480989 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.481481 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.481649 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.485047 (+ 3398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:11.485052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:11.485055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:11.485084 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:11.487586 (+ 2502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.488254 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.488263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.488265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.488414 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.488844 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.488854 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.488855 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.489088 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:11.489243 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:11.489898 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 15:07:11.490095 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":223,"threads_started":1}
I20260925 15:07:11.491340 7567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.474151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37347 (local address 127.4.205.193:57313)
0925 15:07:11.474863 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:11.474903 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:11.476489 (+ 1586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:11.477146 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:11.477154 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:11.477668 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:11.478204 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.478219 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.479167 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.479170 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.480849 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.480864 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.481719 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.481727 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.481938 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.482570 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:11.482595 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:11.484838 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:11.487720 (+ 2882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.487730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.487744 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.488137 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.488521 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.488527 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.488532 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.488713 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.489210 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:11.489217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:11.489515 (+ 298us) client_negotiation.cc:769] Sending connection context
0925 15:07:11.489819 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:07:11.490279 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":252,"threads_started":1}
I20260925 15:07:11.492749 7564 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37347
I20260925 15:07:11.493029 7564 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:11.493868 7564 heartbeater.cc:511] Master 127.4.205.254:37347 requested a full tablet report, sending...
I20260925 15:07:11.496204 7375 ts_manager.cc:194] Registered new tserver with Master: 83c125bb1b844fbf8bef37e3e7215e76 (127.4.205.193:41551)
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:07:11.498390 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:57313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:11.521196 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:11.816291 7570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:11.816542 7570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:11.816685 7570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:11.823551 7570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:11.823674 7570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:11.823854 7570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:11.832875 7570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:11.834062 7570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:11.835322 7570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:11.847525 7575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.847617 7576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:11.847571 7578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:11.849210 7570 server_base.cc:1034] running on GCE node
I20260925 15:07:11.849814 7570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:11.850672 7570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:11.851891 7570 hybrid_clock.cc:648] HybridClock initialized: now 1790348831851842 us; error 40 us; skew 500 ppm
I20260925 15:07:11.852169 7570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:11 dist-test-slave-64fx krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:11.854959 7570 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:11.856350 7570 webserver.cc:460] Webserver started at http://127.4.205.194:36835/ using document root <none> and password file <none>
I20260925 15:07:11.856989 7570 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:11.857128 7570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:11.857420 7570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:11.860121 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "36f85655441641b38f72835bd1772be8"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "9bec2c3356bfef99d2180f463f27c38c"
server_key_iv: "148c3abfe1f00e65516b72c3435667d9"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.860873 7570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "36f85655441641b38f72835bd1772be8"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "9bec2c3356bfef99d2180f463f27c38c"
server_key_iv: "148c3abfe1f00e65516b72c3435667d9"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.865685 7570 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.002s
I20260925 15:07:11.869207 7585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.870642 7570 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:11.870813 7570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "36f85655441641b38f72835bd1772be8"
format_stamp: "Formatted at 2026-09-25 15:07:11 on dist-test-slave-64fx"
server_key: "9bec2c3356bfef99d2180f463f27c38c"
server_key_iv: "148c3abfe1f00e65516b72c3435667d9"
server_key_version: "encryptionkey@0"
I20260925 15:07:11.870999 7570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:11.882241 7570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:11.885462 7570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:11.885756 7570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:11.886606 7570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:11.887693 7570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:11.887784 7570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.887919 7570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:11.887984 7570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:11.919291 7570 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:46341
I20260925 15:07:11.919459 7697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:46341 every 8 connection(s)
I20260925 15:07:11.920492 7570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:11.926448 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7570
I20260925 15:07:11.926647 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:11.927175 4919 external_mini_cluster.cc:1428] Setting key b1c606197c95c5b3f832256c150de9a6
Sep 25 15:07:11 dist-test-slave-64fx krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348831, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:11.941780 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.922792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:33349 (local address 127.4.205.254:37347)
0925 15:07:11.922989 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:11.922995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:11.924091 (+ 1096us) server_negotiation.cc:407] Connection header received
0925 15:07:11.925809 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:11.925814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:11.925875 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:11.925958 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:11.927877 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.928757 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.931242 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.931878 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.931996 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.936346 (+ 4350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:11.936351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:11.936354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:11.936384 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:11.938965 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.939633 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.939643 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.939645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.939768 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:11.940175 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:11.940181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:11.940186 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:11.940427 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:11.940592 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:11.941196 (+ 604us) server_negotiation.cc:299] Negotiation successful
0925 15:07:11.941427 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:07:11.942790 7700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:11.923291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37347 (local address 127.4.205.194:33349)
0925 15:07:11.923942 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:11.923982 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:11.925519 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:11.926176 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:11.926189 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:11.926917 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:11.927684 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.927712 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.928985 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.928991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.931062 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:11.931079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:11.932100 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:11.932109 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:11.932426 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:11.933184 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:11.933214 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:11.936144 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:11.939100 (+ 2956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.939109 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.939123 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.939511 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.939873 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:11.939880 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:11.939885 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:11.940058 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:11.940549 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:11.940556 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:11.940846 (+ 290us) client_negotiation.cc:769] Sending connection context
0925 15:07:11.941079 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 15:07:11.941590 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":260,"threads_started":1}
I20260925 15:07:11.944152 7698 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37347
I20260925 15:07:11.944443 7698 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:11.945055 7698 heartbeater.cc:511] Master 127.4.205.254:37347 requested a full tablet report, sending...
I20260925 15:07:11.946404 7375 ts_manager.cc:194] Registered new tserver with Master: 36f85655441641b38f72835bd1772be8 (127.4.205.194:46341)
I20260925 15:07:11.947165 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:33349
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:11.970546 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:12.266571 7703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:12.266848 7703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:12.267010 7703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:12.273988 7703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:12.274168 7703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:12.274330 7703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:12.282960 7703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37347
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:12.284094 7703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:12.285246 7703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:12.297286 7708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.297365 7709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:12.297334 7711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:12.298132 7703 server_base.cc:1034] running on GCE node
I20260925 15:07:12.298638 7703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:12.299576 7703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:12.300763 7703 hybrid_clock.cc:648] HybridClock initialized: now 1790348832300725 us; error 31 us; skew 500 ppm
I20260925 15:07:12.301038 7703 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:12 dist-test-slave-64fx krb5kdc[7330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:12.303800 7703 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:12.305078 7703 webserver.cc:460] Webserver started at http://127.4.205.195:35633/ using document root <none> and password file <none>
I20260925 15:07:12.305758 7703 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:12.305891 7703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:12.306170 7703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:12.308660 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "86a8d4571b3b444495c215b921f06d19"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-64fx"
server_key: "b2aa8e399bd4b3ff4feceb818cd747c3"
server_key_iv: "ff01eeace25aaf24c85e13dd1c0dda85"
server_key_version: "encryptionkey@0"
I20260925 15:07:12.309376 7703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "86a8d4571b3b444495c215b921f06d19"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-64fx"
server_key: "b2aa8e399bd4b3ff4feceb818cd747c3"
server_key_iv: "ff01eeace25aaf24c85e13dd1c0dda85"
server_key_version: "encryptionkey@0"
I20260925 15:07:12.314029 7703 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:07:12.317049 7718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.318243 7703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:12.318394 7703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "86a8d4571b3b444495c215b921f06d19"
format_stamp: "Formatted at 2026-09-25 15:07:12 on dist-test-slave-64fx"
server_key: "b2aa8e399bd4b3ff4feceb818cd747c3"
server_key_iv: "ff01eeace25aaf24c85e13dd1c0dda85"
server_key_version: "encryptionkey@0"
I20260925 15:07:12.318562 7703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:12.333351 7703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:12.336897 7703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:12.337168 7703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:12.338059 7703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:12.339221 7703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:12.339316 7703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.339457 7703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:12.339525 7703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:12.370347 7703 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:41341
I20260925 15:07:12.370422 7830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:41341 every 8 connection(s)
I20260925 15:07:12.371644 7703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:12.373993 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7703
I20260925 15:07:12.374192 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:12.374706 4919 external_mini_cluster.cc:1428] Setting key 9880a413b1fe99d565c6c1aba6fd6de9
Sep 25 15:07:12 dist-test-slave-64fx krb5kdc[7330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348832, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:12.390585 7566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.374659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:55023 (local address 127.4.205.254:37347)
0925 15:07:12.374858 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.374864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.376129 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 15:07:12.377947 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.377951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.378010 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.378086 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.379409 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.380093 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.381506 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.381978 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.382067 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.385089 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:12.385093 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:12.385096 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:12.385125 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:12.387734 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.388370 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.388377 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.388384 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.388547 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:12.389024 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:12.389032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:12.389036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:12.389397 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:12.389530 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:12.390183 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 15:07:12.390366 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:07:12.391451 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:12.375201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37347 (local address 127.4.205.195:55023)
0925 15:07:12.375985 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:12.376027 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:12.377728 (+ 1701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:12.378270 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:12.378277 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:12.378712 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:12.379240 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.379256 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.380222 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.380226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.381394 (+ 1168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:12.381405 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:12.382141 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:12.382145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:12.382323 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:12.382906 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:12.382926 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:12.384924 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:12.387876 (+ 2952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.387882 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.387891 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.388235 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.388659 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:12.388662 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:12.388664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:12.388914 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:12.389526 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:12.389535 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:12.389877 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 15:07:12.390107 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:07:12.390453 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":328,"threads_started":1}
I20260925 15:07:12.392670 7831 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37347
I20260925 15:07:12.392951 7831 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:12.393627 7831 heartbeater.cc:511] Master 127.4.205.254:37347 requested a full tablet report, sending...
I20260925 15:07:12.394836 7375 ts_manager.cc:194] Registered new tserver with Master: 86a8d4571b3b444495c215b921f06d19 (127.4.205.195:41341)
I20260925 15:07:12.395928 7375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:55023
I20260925 15:07:12.401901 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:07:12.429840 7566 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:12.427594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.4.205.254:37347)
0925 15:07:12.427830 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:12.427840 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:12.428167 (+ 327us) server_negotiation.cc:407] Connection header received
0925 15:07:12.428303 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:12.428306 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:12.428366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:12.428446 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:12.429536 (+ 1090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34362: BlockingRecv error: recv got EOF from 127.0.0.1:34362 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:07:12.432405 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7436
W20260925 15:07:12.434836 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7436: Not authorized: Client connection negotiation failed: client connection to 127.4.205.193:41551: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:07:12.434989 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7436
I20260925 15:07:12.449622 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7570
W20260925 15:07:12.451900 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7570: Not authorized: Client connection negotiation failed: client connection to 127.4.205.194:46341: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:07:12.452040 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7570
I20260925 15:07:12.466754 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7703
W20260925 15:07:12.469234 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7703: Not authorized: Client connection negotiation failed: client connection to 127.4.205.195:41341: server requires authentication, but client does not have Kerberos credentials available
I20260925 15:07:12.469347 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7703
I20260925 15:07:12.484781 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7343
I20260925 15:07:12.485920 7405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:12.583194 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7343
2026-09-25T15:07:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3257 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:12 dist-test-slave-64fx krb5kdc[7848](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:12 dist-test-slave-64fx krb5kdc[7848](info): set up 2 sockets
Sep 25 15:07:12 dist-test-slave-64fx krb5kdc[7848](info): commencing operation
krb5kdc: starting...
W20260925 15:07:13.617923 4919 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:13 dist-test-slave-64fx krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:13.792615 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:34893
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:34893
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:14.095633 7861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:14.095896 7861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:14.096064 7861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:14.102885 7861 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:14.102974 7861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:14.103109 7861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:14.103199 7861 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:14.103288 7861 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:14.112202 7861 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:34893
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:34893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:14.113319 7861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:14.114516 7861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:14.126148 7866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:14.126236 7867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:14.126187 7869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:14.127153 7861 server_base.cc:1034] running on GCE node
I20260925 15:07:14.127912 7861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:14.129102 7861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:14.130321 7861 hybrid_clock.cc:648] HybridClock initialized: now 1790348834130265 us; error 66 us; skew 500 ppm
I20260925 15:07:14.130636 7861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:14 dist-test-slave-64fx krb5kdc[7848](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348834, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:14.138592 7861 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:14.140527 7861 webserver.cc:460] Webserver started at http://127.4.205.254:34883/ using document root <none> and password file <none>
I20260925 15:07:14.141412 7861 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:14.141599 7861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:14.141983 7861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:14.145228 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "5bf00ec0b2724567896a312314f97646"
format_stamp: "Formatted at 2026-09-25 15:07:14 on dist-test-slave-64fx"
server_key: "e3d7322a12518a69d840ed335b98c73d"
server_key_iv: "0287eece4bcd0f5d0d6517bfe9ee7684"
server_key_version: "encryptionkey@0"
I20260925 15:07:14.146162 7861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "5bf00ec0b2724567896a312314f97646"
format_stamp: "Formatted at 2026-09-25 15:07:14 on dist-test-slave-64fx"
server_key: "e3d7322a12518a69d840ed335b98c73d"
server_key_iv: "0287eece4bcd0f5d0d6517bfe9ee7684"
server_key_version: "encryptionkey@0"
I20260925 15:07:14.150956 7861 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:07:14.154001 7876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:14.155128 7861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 15:07:14.155297 7861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "5bf00ec0b2724567896a312314f97646"
format_stamp: "Formatted at 2026-09-25 15:07:14 on dist-test-slave-64fx"
server_key: "e3d7322a12518a69d840ed335b98c73d"
server_key_iv: "0287eece4bcd0f5d0d6517bfe9ee7684"
server_key_version: "encryptionkey@0"
I20260925 15:07:14.155539 7861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:14.168502 7861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:14.172236 7861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:14.172504 7861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:14.189045 7861 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:34893
I20260925 15:07:14.189077 7927 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:34893 every 8 connection(s)
I20260925 15:07:14.190407 7861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:14.193809 7928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:14.196781 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7861
I20260925 15:07:14.196967 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:14.197778 4919 external_mini_cluster.cc:1428] Setting key c9fd1800387ba043f26ac71971b2ed17
I20260925 15:07:14.205807 7928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646: Bootstrap starting.
I20260925 15:07:14.209231 7928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:14.210481 7928 log.cc:824] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:14 dist-test-slave-64fx krb5kdc[7848](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348833, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:14.212955 7928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646: No bootstrap required, opened a new log
I20260925 15:07:14.220013 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:14.199471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55698 (local address 127.4.205.254:34893)
0925 15:07:14.200174 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:14.200184 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:14.200221 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:07:14.201859 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:14.201881 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:14.202241 (+ 360us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:14.202558 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:14.204297 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.205539 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.207282 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.207895 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.208013 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:14.212261 (+ 4248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:14.212268 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:14.212280 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:14.212311 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:14.215994 (+ 3683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:14.216611 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:14.216619 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:14.216627 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:14.216830 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:14.217157 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:14.217162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:14.217166 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:14.217798 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:14.218062 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:14.218360 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:07:14.218783 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":339,"threads_started":1}
I20260925 15:07:14.223011 7928 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } }
I20260925 15:07:14.223368 7928 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:14.223470 7928 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bf00ec0b2724567896a312314f97646, State: Initialized, Role: FOLLOWER
I20260925 15:07:14.223811 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:14.224037 7928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } }
I20260925 15:07:14.224253 7928 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:14.224406 7928 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:14.224562 7928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:14.226629 7928 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } }
I20260925 15:07:14.227082 7928 leader_election.cc:302] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bf00ec0b2724567896a312314f97646; no voters:
I20260925 15:07:14.227631 7928 leader_election.cc:288] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:14.227758 7933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:14.229383 7928 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:14.230950 7933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [term 1 LEADER]: Becoming Leader. State: Replica: 5bf00ec0b2724567896a312314f97646, State: Running, Role: LEADER
I20260925 15:07:14.231614 7933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } }
I20260925 15:07:14.235590 7935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bf00ec0b2724567896a312314f97646. Latest consensus state: current_term: 1 leader_uuid: "5bf00ec0b2724567896a312314f97646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } } }
I20260925 15:07:14.235939 7935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:14.235684 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bf00ec0b2724567896a312314f97646" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bf00ec0b2724567896a312314f97646" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34893 } } }
I20260925 15:07:14.236048 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:14.236799 7949 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:14.252005 7949 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:14.260743 7949 catalog_manager.cc:1347] Generated new cluster ID: bf3f79e9c0594aada4729e27b300711c
I20260925 15:07:14.260928 7949 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:14.268694 7931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:14.261267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.4.205.254:34893)
0925 15:07:14.261494 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:14.261503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:14.262095 (+ 592us) server_negotiation.cc:407] Connection header received
0925 15:07:14.262304 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:14.262310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:14.262390 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:14.262512 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:14.263455 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.264211 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.265718 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:14.266161 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:14.266293 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:14.267336 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:14.267343 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:14.267377 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:14.267918 (+ 541us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:14.267973 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:14.267997 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:14.268108 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:14.268241 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:07:14.268330 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:07:14.286464 7949 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:14.288141 7949 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:14.299914 7949 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bf00ec0b2724567896a312314f97646: Generated new TSK 0
I20260925 15:07:14.300820 7949 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:14.312537 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7861
I20260925 15:07:14.313690 7923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:14.403172 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7861
2026-09-25T15:07:14Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1815 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:14 dist-test-slave-64fx krb5kdc[7962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:14 dist-test-slave-64fx krb5kdc[7962](info): set up 2 sockets
Sep 25 15:07:14 dist-test-slave-64fx krb5kdc[7962](info): commencing operation
krb5kdc: starting...
W20260925 15:07:15.453910 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:15 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:15.636232 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:46197
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:46197
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:15.943276 7975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:15.943540 7975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:15.943708 7975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:15.950636 7975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:15.950769 7975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:15.950857 7975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:15.950958 7975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:15.951045 7975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:15.960338 7975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:46197
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:46197
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:15.961629 7975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:15.962850 7975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:15.974583 7980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:15.974514 7981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:15.974551 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:15.975652 7975 server_base.cc:1034] running on GCE node
I20260925 15:07:15.976464 7975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:15.977808 7975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:15.979017 7975 hybrid_clock.cc:648] HybridClock initialized: now 1790348835978988 us; error 26 us; skew 500 ppm
I20260925 15:07:15.979365 7975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:15 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:15.982321 7975 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:15.983750 7975 webserver.cc:460] Webserver started at http://127.4.205.254:36427/ using document root <none> and password file <none>
I20260925 15:07:15.984396 7975 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:15.984537 7975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:15.984826 7975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:15.987640 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "77883a3c5ef3434fbd7d85d9b116559d"
format_stamp: "Formatted at 2026-09-25 15:07:15 on dist-test-slave-64fx"
server_key: "fd2cd2252c52133aeb538a113a8d3aed"
server_key_iv: "cc1410d05f9de689412712eafbb5a10b"
server_key_version: "encryptionkey@0"
I20260925 15:07:15.988358 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "77883a3c5ef3434fbd7d85d9b116559d"
format_stamp: "Formatted at 2026-09-25 15:07:15 on dist-test-slave-64fx"
server_key: "fd2cd2252c52133aeb538a113a8d3aed"
server_key_iv: "cc1410d05f9de689412712eafbb5a10b"
server_key_version: "encryptionkey@0"
I20260925 15:07:15.993247 7975 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:07:15.996507 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:15.997836 7975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:07:15.998000 7975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "77883a3c5ef3434fbd7d85d9b116559d"
format_stamp: "Formatted at 2026-09-25 15:07:15 on dist-test-slave-64fx"
server_key: "fd2cd2252c52133aeb538a113a8d3aed"
server_key_iv: "cc1410d05f9de689412712eafbb5a10b"
server_key_version: "encryptionkey@0"
I20260925 15:07:15.998179 7975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:16.030757 7975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:16.034678 7975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:16.034970 7975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:16.053260 7975 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:46197
I20260925 15:07:16.053315 8041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:46197 every 8 connection(s)
I20260925 15:07:16.054560 7975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:16.058310 8042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:16.063292 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 7975
I20260925 15:07:16.063479 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:16.064114 4919 external_mini_cluster.cc:1428] Setting key d706f80f06783910c179a03b10a710c7
I20260925 15:07:16.069491 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d: Bootstrap starting.
I20260925 15:07:16.073071 8042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:16.074188 8042 log.cc:824] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:16.076997 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d: No bootstrap required, opened a new log
Sep 25 15:07:16 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:16.087065 8042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } }
I20260925 15:07:16.087462 8045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.066066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56488 (local address 127.4.205.254:46197)
0925 15:07:16.066888 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.066906 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.066963 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:07:16.069196 (+ 2233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.069222 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.069712 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.070119 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:16.071322 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.073295 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.074620 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.075458 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.075634 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.079301 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:16.079310 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:16.079326 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:16.079381 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:16.083277 (+ 3896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.083847 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.083855 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.083880 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.084097 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.084460 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.084470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.084474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.085150 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:16.085466 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.085786 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.086191 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":368,"threads_started":1}
I20260925 15:07:16.087488 8042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:16.087582 8042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77883a3c5ef3434fbd7d85d9b116559d, State: Initialized, Role: FOLLOWER
I20260925 15:07:16.088104 8042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } }
I20260925 15:07:16.088380 8042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:16.088531 8042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:16.088706 8042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:16.091046 8042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } }
I20260925 15:07:16.091588 8042 leader_election.cc:302] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77883a3c5ef3434fbd7d85d9b116559d; no voters:
I20260925 15:07:16.092222 8042 leader_election.cc:288] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:16.092317 8047 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:16.093055 8047 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [term 1 LEADER]: Becoming Leader. State: Replica: 77883a3c5ef3434fbd7d85d9b116559d, State: Running, Role: LEADER
I20260925 15:07:16.093587 8047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } }
I20260925 15:07:16.094520 8042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:16.096192 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77883a3c5ef3434fbd7d85d9b116559d. Latest consensus state: current_term: 1 leader_uuid: "77883a3c5ef3434fbd7d85d9b116559d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } } }
I20260925 15:07:16.096522 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:16.098270 8048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77883a3c5ef3434fbd7d85d9b116559d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77883a3c5ef3434fbd7d85d9b116559d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46197 } } }
I20260925 15:07:16.098498 8052 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:16.098568 8048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:16.104593 8052 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:16.114774 8052 catalog_manager.cc:1347] Generated new cluster ID: 8619415e3daf4b598e39f4dcb5818ec1
I20260925 15:07:16.114871 8052 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:16.181087 8052 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:16.182499 8052 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:16.188907 8052 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77883a3c5ef3434fbd7d85d9b116559d: Generated new TSK 0
I20260925 15:07:16.189764 8052 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:16.245406 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:16.560647 8068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:16.561023 8068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:16.561210 8068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:16.568356 8068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:16.568521 8068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:16.568714 8068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:16.578586 8068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:16.579859 8068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:16.581121 8068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:16.593921 8073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:16.593956 8076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:16.593875 8074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:16.594760 8068 server_base.cc:1034] running on GCE node
I20260925 15:07:16.595376 8068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:16.596298 8068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:16.597517 8068 hybrid_clock.cc:648] HybridClock initialized: now 1790348836597449 us; error 52 us; skew 500 ppm
I20260925 15:07:16.597816 8068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:16 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348836, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:16.600761 8068 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:16.602249 8068 webserver.cc:460] Webserver started at http://127.4.205.193:42835/ using document root <none> and password file <none>
I20260925 15:07:16.602949 8068 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:16.603078 8068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:16.603350 8068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:16.606099 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "e2a7ac5791ec4d118c1f53b647785d0e"
format_stamp: "Formatted at 2026-09-25 15:07:16 on dist-test-slave-64fx"
server_key: "9503b16ddfbceebb0f71078364b7a6f2"
server_key_iv: "061f994f58170251ee6a12dbb7373565"
server_key_version: "encryptionkey@0"
I20260925 15:07:16.606830 8068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "e2a7ac5791ec4d118c1f53b647785d0e"
format_stamp: "Formatted at 2026-09-25 15:07:16 on dist-test-slave-64fx"
server_key: "9503b16ddfbceebb0f71078364b7a6f2"
server_key_iv: "061f994f58170251ee6a12dbb7373565"
server_key_version: "encryptionkey@0"
I20260925 15:07:16.611579 8068 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:16.614640 8083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:16.615808 8068 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:16.615959 8068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "e2a7ac5791ec4d118c1f53b647785d0e"
format_stamp: "Formatted at 2026-09-25 15:07:16 on dist-test-slave-64fx"
server_key: "9503b16ddfbceebb0f71078364b7a6f2"
server_key_iv: "061f994f58170251ee6a12dbb7373565"
server_key_version: "encryptionkey@0"
I20260925 15:07:16.616144 8068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:16.641288 8068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:16.645234 8068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:16.645529 8068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:16.646466 8068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:16.647717 8068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:16.647786 8068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:16.647943 8068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:16.648006 8068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:16.680290 8068 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:42179
I20260925 15:07:16.680343 8195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:42179 every 8 connection(s)
I20260925 15:07:16.681805 8068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:16.682978 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8068
I20260925 15:07:16.683171 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:16.683997 4919 external_mini_cluster.cc:1428] Setting key bf299b47f596c491255b2da94e9d8cd8
Sep 25 15:07:16 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348836, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:07:16.704589 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.684701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42315 (local address 127.4.205.254:46197)
0925 15:07:16.685167 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:16.685175 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:16.686270 (+ 1095us) server_negotiation.cc:407] Connection header received
0925 15:07:16.688033 (+ 1763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:16.688041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:16.688128 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:16.688280 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:16.690037 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.691091 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.693652 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.694082 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.694172 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.698222 (+ 4050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:16.698231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:16.698239 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:16.698283 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:16.701717 (+ 3434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.702417 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.702426 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.702432 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.702585 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:16.703081 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:16.703089 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:16.703093 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:16.703455 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:16.703562 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:16.704196 (+ 634us) server_negotiation.cc:299] Negotiation successful
0925 15:07:16.704351 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":844,"threads_started":1}
I20260925 15:07:16.705674 8199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:16.685271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46197 (local address 127.4.205.193:42315)
0925 15:07:16.686103 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:16.686155 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:16.687777 (+ 1622us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:16.688590 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:16.688598 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:16.689234 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:16.689876 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.689895 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.691243 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.691250 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.693491 (+ 2241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:16.693513 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:16.694294 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:16.694303 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:16.694619 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:16.695341 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:16.695372 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:16.698001 (+ 2629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:16.701868 (+ 3867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:16.701880 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:16.701891 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:16.702281 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:16.702699 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:16.702701 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:16.702703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:16.702952 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:16.703560 (+ 608us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:16.703565 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:16.703842 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 15:07:16.704094 (+ 252us) client_negotiation.cc:240] Negotiation successful
0925 15:07:16.704493 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
I20260925 15:07:16.707207 8196 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46197
I20260925 15:07:16.707512 8196 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:16.708333 8196 heartbeater.cc:511] Master 127.4.205.254:46197 requested a full tablet report, sending...
I20260925 15:07:16.710629 8007 ts_manager.cc:194] Registered new tserver with Master: e2a7ac5791ec4d118c1f53b647785d0e (127.4.205.193:42179)
I20260925 15:07:16.712212 8007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:42315
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:16.728420 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:17.026782 8202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:17.027022 8202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:17.027107 8202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:17.033766 8202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:17.033849 8202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:17.033946 8202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:17.043322 8202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:17.044318 8202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:17.045430 8202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:17.057201 8207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:17.057231 8208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:17.057907 8210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:17.062130 8202 server_base.cc:1034] running on GCE node
I20260925 15:07:17.062705 8202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:17.063593 8202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:17.064803 8202 hybrid_clock.cc:648] HybridClock initialized: now 1790348837064762 us; error 38 us; skew 500 ppm
I20260925 15:07:17.065089 8202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:17.068043 8202 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:17.069268 8202 webserver.cc:460] Webserver started at http://127.4.205.194:42137/ using document root <none> and password file <none>
I20260925 15:07:17.069948 8202 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:17.070082 8202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:17.070355 8202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:17.072830 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "72b260bf970f4d528ab814de60e14327"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "1c37d35d0d078bc46240523ddbab2f88"
server_key_iv: "241190be43abffbabf38aca52e28e50d"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.073516 8202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "72b260bf970f4d528ab814de60e14327"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "1c37d35d0d078bc46240523ddbab2f88"
server_key_iv: "241190be43abffbabf38aca52e28e50d"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.078167 8202 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:07:17.081311 8217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.082496 8202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:17.082666 8202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "72b260bf970f4d528ab814de60e14327"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "1c37d35d0d078bc46240523ddbab2f88"
server_key_iv: "241190be43abffbabf38aca52e28e50d"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.082849 8202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:17.106637 8202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:17.110041 8202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:17.110314 8202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:17.111140 8202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:17.112255 8202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:17.112346 8202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.112473 8202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:17.112534 8202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.144182 8202 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:34865
I20260925 15:07:17.144222 8329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:34865 every 8 connection(s)
I20260925 15:07:17.145370 8202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:17.153398 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8202
I20260925 15:07:17.153618 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:17.154035 4919 external_mini_cluster.cc:1428] Setting key 361df977272da1ee486a7817f18105a2
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:17.165282 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.147740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:52619 (local address 127.4.205.254:46197)
0925 15:07:17.147928 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.147934 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.149081 (+ 1147us) server_negotiation.cc:407] Connection header received
0925 15:07:17.150806 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.150810 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.150873 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.150955 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.152279 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.153089 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.154855 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.155513 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.155632 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.159852 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.159859 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.159865 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.159900 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.162484 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.163156 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.163162 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.163167 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.163296 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.163700 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.163706 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.163710 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.163957 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.164102 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.164738 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.164952 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:07:17.166278 8332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.148273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46197 (local address 127.4.205.194:52619)
0925 15:07:17.148925 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:17.148959 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:17.150561 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:17.151137 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:17.151144 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:17.151600 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:17.152131 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:17.152146 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.153249 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.153252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:17.154712 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:17.154734 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.155783 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.155794 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:17.156161 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.157093 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:17.157125 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:17.159646 (+ 2521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:17.162616 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:17.162626 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:17.162642 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:17.163034 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:17.163400 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:17.163406 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:17.163411 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:17.163586 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:17.164075 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:17.164082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:17.164401 (+ 319us) client_negotiation.cc:769] Sending connection context
0925 15:07:17.164655 (+ 254us) client_negotiation.cc:240] Negotiation successful
0925 15:07:17.165122 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":249,"threads_started":1}
I20260925 15:07:17.167603 8330 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46197
I20260925 15:07:17.167905 8330 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:17.168756 8330 heartbeater.cc:511] Master 127.4.205.254:46197 requested a full tablet report, sending...
I20260925 15:07:17.170351 8007 ts_manager.cc:194] Registered new tserver with Master: 72b260bf970f4d528ab814de60e14327 (127.4.205.194:34865)
I20260925 15:07:17.171587 8007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:52619
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:17.194605 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:17.496222 8335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:17.496469 8335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:17.496609 8335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:17.503791 8335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:17.503925 8335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:17.504115 8335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:17.513162 8335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:17.514353 8335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:17.515558 8335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:17.527384 8343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:17.527462 8341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:17.527496 8340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:17.528239 8335 server_base.cc:1034] running on GCE node
I20260925 15:07:17.528791 8335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:17.529645 8335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:17.530833 8335 hybrid_clock.cc:648] HybridClock initialized: now 1790348837530797 us; error 33 us; skew 500 ppm
I20260925 15:07:17.531112 8335 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:17.533860 8335 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:17.535081 8335 webserver.cc:460] Webserver started at http://127.4.205.195:36737/ using document root <none> and password file <none>
I20260925 15:07:17.535696 8335 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:17.535859 8335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:17.536134 8335 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:17.538810 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "bf8eee250c4e4f93b5ce698f9f88d848"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "e4060bfbd5c10b2125fbc4a3aeab40d0"
server_key_iv: "7dd60cab90bc35a4e87c312e03aa1d5a"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.539532 8335 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "bf8eee250c4e4f93b5ce698f9f88d848"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "e4060bfbd5c10b2125fbc4a3aeab40d0"
server_key_iv: "7dd60cab90bc35a4e87c312e03aa1d5a"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.544143 8335 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:07:17.547194 8350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.548370 8335 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:17.548529 8335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "bf8eee250c4e4f93b5ce698f9f88d848"
format_stamp: "Formatted at 2026-09-25 15:07:17 on dist-test-slave-64fx"
server_key: "e4060bfbd5c10b2125fbc4a3aeab40d0"
server_key_iv: "7dd60cab90bc35a4e87c312e03aa1d5a"
server_key_version: "encryptionkey@0"
I20260925 15:07:17.548710 8335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:17.568084 8335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:17.571580 8335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:17.571863 8335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:17.572674 8335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:17.573853 8335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:17.573921 8335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.573997 8335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:17.574040 8335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:17.605598 8335 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:41919
I20260925 15:07:17.605687 8462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:41919 every 8 connection(s)
I20260925 15:07:17.606840 8335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:17.609756 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8335
I20260925 15:07:17.609934 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:17.610365 4919 external_mini_cluster.cc:1428] Setting key ce2c21d1ffeb210b0fd1ee8984816afa
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:17.625681 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.609323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:56781 (local address 127.4.205.254:46197)
0925 15:07:17.609727 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.609734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.610943 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 15:07:17.613019 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.613023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.613104 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.613219 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.614629 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.615392 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.616856 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.617257 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.617387 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.620548 (+ 3161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.620556 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.620589 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.620632 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.622903 (+ 2271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.623566 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.623572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.623578 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.623687 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.624091 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.624097 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.624101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.624358 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.624544 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.625118 (+ 574us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.625316 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280}
I20260925 15:07:17.626515 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.609982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46197 (local address 127.4.205.195:56781)
0925 15:07:17.610781 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:17.610829 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:17.612700 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:17.613442 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:17.613450 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:17.613969 (+ 519us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:17.614482 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:17.614497 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.615532 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.615545 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:17.616720 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:17.616731 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.617463 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.617472 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:17.617739 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.618292 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:17.618317 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:17.620352 (+ 2035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:17.623032 (+ 2680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:17.623041 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:17.623056 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:17.623445 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:17.623794 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:17.623800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:17.623805 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:17.623978 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:17.624475 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:17.624482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:17.624796 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 15:07:17.625032 (+ 236us) client_negotiation.cc:240] Negotiation successful
0925 15:07:17.625507 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":313,"threads_started":1}
I20260925 15:07:17.627712 8463 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46197
I20260925 15:07:17.628010 8463 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:17.628680 8463 heartbeater.cc:511] Master 127.4.205.254:46197 requested a full tablet report, sending...
I20260925 15:07:17.630034 8007 ts_manager.cc:194] Registered new tserver with Master: bf8eee250c4e4f93b5ce698f9f88d848 (127.4.205.195:41919)
I20260925 15:07:17.631002 8007 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:56781
I20260925 15:07:17.636984 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:17.680377 8198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.667589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56500 (local address 127.4.205.254:46197)
0925 15:07:17.667788 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.667797 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.668205 (+ 408us) server_negotiation.cc:407] Connection header received
0925 15:07:17.668422 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.668426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.668483 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.668583 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.669930 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.670627 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.671793 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.672163 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.672293 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.675884 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.675889 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.675892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.675921 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.678261 (+ 2340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.678763 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.678767 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.678769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.678870 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.679144 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.679147 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.679149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.679445 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.679680 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.679848 (+ 168us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.680051 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 15:07:17.680926 8007 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56500
I20260925 15:07:17.682801 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8068
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:17.696570 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.683815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33368 (local address 127.4.205.193:42179)
0925 15:07:17.684434 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.684444 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.684470 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:07:17.684532 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.684540 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.684760 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.684898 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.686021 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.687040 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.688049 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.688541 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.688625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.691999 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.692008 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.692019 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.692044 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.694427 (+ 2383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.694981 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.694987 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.694989 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.695089 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.695358 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.695361 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.695363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.695694 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.695994 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.696132 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.696299 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":304,"threads_started":1}
W20260925 15:07:17.697714 8191 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:33368
W20260925 15:07:17.698699 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8068: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:17.698839 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8068
I20260925 15:07:17.713670 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8202
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:17.730958 8475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.714391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32958 (local address 127.4.205.194:34865)
0925 15:07:17.714919 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.714926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.714947 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:17.714997 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.715003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.715183 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.715289 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.716323 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.717250 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.718476 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.718975 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.719099 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.726743 (+ 7644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.726753 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.726765 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.726793 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.728957 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.729435 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.729441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.729443 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.729544 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.729864 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.729867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.729869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.730130 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.730324 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.730525 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.730703 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260925 15:07:17.732048 8325 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32958
W20260925 15:07:17.732972 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8202: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:17.733090 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8202
I20260925 15:07:17.747597 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8335
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[7962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348837, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:17.761070 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:17.748278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36622 (local address 127.4.205.195:41919)
0925 15:07:17.748833 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:17.748838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:17.748863 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:17.748911 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:17.748916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:17.749085 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:17.749192 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:17.750323 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.751213 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.752335 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:17.752823 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:17.752910 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:17.756592 (+ 3682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:17.756601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:17.756613 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:17.756639 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:17.759058 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.759585 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.759590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.759592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.759687 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:17.759953 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:17.759956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:17.759957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:17.760247 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:17.760463 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:17.760667 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:07:17.760823 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":289,"threads_started":1}
W20260925 15:07:17.762259 8458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36622
W20260925 15:07:17.763202 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8335: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:07:17.763334 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8335
I20260925 15:07:17.777383 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 7975
I20260925 15:07:17.778314 8037 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:17.889674 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 7975
2026-09-25T15:07:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3492 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[8480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[8480](info): set up 2 sockets
Sep 25 15:07:17 dist-test-slave-64fx krb5kdc[8480](info): commencing operation
krb5kdc: starting...
W20260925 15:07:18.945926 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:19 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:19.130836 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42971
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:42971
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:19.431568 8493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:19.431842 8493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:19.431931 8493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:19.438807 8493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:19.438905 8493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:19.438958 8493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:19.439013 8493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:19.439056 8493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:19.448267 8493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:42971
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42971
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:19.449425 8493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:19.450609 8493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:19.461632 8498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.461711 8501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:19.461632 8499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:19.463925 8493 server_base.cc:1034] running on GCE node
I20260925 15:07:19.464665 8493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:19.466050 8493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:19.467284 8493 hybrid_clock.cc:648] HybridClock initialized: now 1790348839467240 us; error 54 us; skew 500 ppm
I20260925 15:07:19.467607 8493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:19 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:19.477823 8493 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:19.479280 8493 webserver.cc:460] Webserver started at http://127.4.205.254:45673/ using document root <none> and password file <none>
I20260925 15:07:19.479947 8493 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:19.480075 8493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:19.480446 8493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:19.483309 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "740f5ac88ae94446a5d07ddb365a35a7"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-64fx"
server_key: "68a57174821c7aa26189b59c2ceec6c8"
server_key_iv: "62f8c661db043ab14f20fd6e537f727c"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.484025 8493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "740f5ac88ae94446a5d07ddb365a35a7"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-64fx"
server_key: "68a57174821c7aa26189b59c2ceec6c8"
server_key_iv: "62f8c661db043ab14f20fd6e537f727c"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.488797 8493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:07:19.491842 8508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:19.493046 8493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:07:19.493211 8493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "740f5ac88ae94446a5d07ddb365a35a7"
format_stamp: "Formatted at 2026-09-25 15:07:19 on dist-test-slave-64fx"
server_key: "68a57174821c7aa26189b59c2ceec6c8"
server_key_iv: "62f8c661db043ab14f20fd6e537f727c"
server_key_version: "encryptionkey@0"
I20260925 15:07:19.493391 8493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:19.507910 8493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:19.511538 8493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:19.511806 8493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:19.529345 8493 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:42971
I20260925 15:07:19.529349 8559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:42971 every 8 connection(s)
I20260925 15:07:19.530702 8493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:19.534070 8560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:19.537179 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8493
I20260925 15:07:19.537393 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:19.538180 4919 external_mini_cluster.cc:1428] Setting key 428f5b5ea83650884ba39fb606c4ece2
I20260925 15:07:19.546372 8560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7: Bootstrap starting.
I20260925 15:07:19.550010 8560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:19.551288 8560 log.cc:824] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:19 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348839, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:19.554301 8560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7: No bootstrap required, opened a new log
I20260925 15:07:19.560393 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:19.540019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.4.205.254:42971)
0925 15:07:19.540712 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:19.540724 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:19.540765 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:07:19.542428 (+ 1663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:19.542449 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:19.542824 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:19.543118 (+ 294us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:19.544813 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.546120 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.547775 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:19.548430 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:19.548550 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:19.553319 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:19.553326 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:19.553340 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:19.553371 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:19.556385 (+ 3014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:19.557311 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:19.557319 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:19.557325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:19.557475 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:19.557904 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:19.557907 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:19.557909 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:19.558408 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:19.558611 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:19.559049 (+ 438us) server_negotiation.cc:299] Negotiation successful
0925 15:07:19.559393 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":337,"threads_started":1}
I20260925 15:07:19.563452 8560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } }
I20260925 15:07:19.563791 8560 raft_consensus.cc:374] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:19.563902 8560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 740f5ac88ae94446a5d07ddb365a35a7, State: Initialized, Role: FOLLOWER
I20260925 15:07:19.564416 8560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } }
I20260925 15:07:19.564649 8560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:19.564791 8560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:19.564975 8560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:19.567189 8560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } }
I20260925 15:07:19.567658 8560 leader_election.cc:302] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 740f5ac88ae94446a5d07ddb365a35a7; no voters:
I20260925 15:07:19.568202 8560 leader_election.cc:288] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:19.568323 8565 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:19.569440 8565 raft_consensus.cc:686] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [term 1 LEADER]: Becoming Leader. State: Replica: 740f5ac88ae94446a5d07ddb365a35a7, State: Running, Role: LEADER
I20260925 15:07:19.569988 8565 consensus_queue.cc:237] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } }
I20260925 15:07:19.570385 8560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:19.572464 8566 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "740f5ac88ae94446a5d07ddb365a35a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } } }
I20260925 15:07:19.572829 8566 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:19.573686 8574 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:19.573355 8567 sys_catalog.cc:455] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 740f5ac88ae94446a5d07ddb365a35a7. Latest consensus state: current_term: 1 leader_uuid: "740f5ac88ae94446a5d07ddb365a35a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "740f5ac88ae94446a5d07ddb365a35a7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42971 } } }
I20260925 15:07:19.574338 8567 sys_catalog.cc:458] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:19.586951 8574 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:19.595221 8574 catalog_manager.cc:1347] Generated new cluster ID: 2c11d8c6f3184cd08729bc2655f173d0
I20260925 15:07:19.595345 8574 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:19.613607 8574 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:19.615185 8574 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:19.627516 8574 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 740f5ac88ae94446a5d07ddb365a35a7: Generated new TSK 0
I20260925 15:07:19.628324 8574 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:19.679253 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:20.004575 8587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:20.004801 8587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:20.004902 8587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:20.011654 8587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:20.011777 8587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:20.011880 8587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:20.020905 8587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:20.022012 8587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:20.023216 8587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:20.035837 8592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.035859 8593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.035893 8595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:20.036725 8587 server_base.cc:1034] running on GCE node
I20260925 15:07:20.037281 8587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:20.038205 8587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:20.039410 8587 hybrid_clock.cc:648] HybridClock initialized: now 1790348840039376 us; error 27 us; skew 500 ppm
I20260925 15:07:20.039707 8587 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:20 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:20.042706 8587 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:20.044041 8587 webserver.cc:460] Webserver started at http://127.4.205.193:41973/ using document root <none> and password file <none>
I20260925 15:07:20.044668 8587 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:20.044808 8587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:20.045095 8587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:20.047919 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "e5121ad16ff84f4bb549ed56c9241af6"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "7447f79bfe4a90c08ddacb7563be2519"
server_key_iv: "7c4b39e837ab0d1f1c02679abb6f06ae"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.048645 8587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "e5121ad16ff84f4bb549ed56c9241af6"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "7447f79bfe4a90c08ddacb7563be2519"
server_key_iv: "7c4b39e837ab0d1f1c02679abb6f06ae"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.053418 8587 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:07:20.056545 8602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.057826 8587 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:20.057981 8587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "e5121ad16ff84f4bb549ed56c9241af6"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "7447f79bfe4a90c08ddacb7563be2519"
server_key_iv: "7c4b39e837ab0d1f1c02679abb6f06ae"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.058164 8587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:20.069741 8587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:20.073830 8587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:20.074141 8587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:20.075007 8587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:20.076215 8587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:20.076321 8587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.076442 8587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:20.076522 8587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.106734 8587 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:40191
I20260925 15:07:20.106786 8714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:40191 every 8 connection(s)
I20260925 15:07:20.108059 8587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:20.118376 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8587
I20260925 15:07:20.118620 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:20.119213 4919 external_mini_cluster.cc:1428] Setting key 5e6dddb1d460baeaa7f0e15f49940f33
Sep 25 15:07:20 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:20.127125 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.110636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37033 (local address 127.4.205.254:42971)
0925 15:07:20.111045 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.111050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.112024 (+ 974us) server_negotiation.cc:407] Connection header received
0925 15:07:20.113736 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.113741 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.113829 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.113952 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.115329 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.116057 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.117578 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.118150 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.118286 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.121905 (+ 3619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.121910 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.121913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.121941 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.124283 (+ 2342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.124929 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.124935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.124939 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.125049 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.125430 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.125437 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.125441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.125757 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.125904 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.126525 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.126743 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260925 15:07:20.127946 8718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.111207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42971 (local address 127.4.205.193:37033)
0925 15:07:20.111856 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.111899 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.113460 (+ 1561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.114166 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.114173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.114629 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.115165 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.115180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.116195 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.116203 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.117420 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.117430 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.118345 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.118350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.118603 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.119309 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.119337 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.121693 (+ 2356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.124415 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.124424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.124438 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.124811 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.125150 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.125156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.125161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.125329 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.125881 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.125889 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.126203 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.126444 (+ 241us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.126950 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":248,"threads_started":1}
I20260925 15:07:20.129312 8715 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42971
I20260925 15:07:20.129642 8715 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:20.130477 8715 heartbeater.cc:511] Master 127.4.205.254:42971 requested a full tablet report, sending...
I20260925 15:07:20.132480 8525 ts_manager.cc:194] Registered new tserver with Master: e5121ad16ff84f4bb549ed56c9241af6 (127.4.205.193:40191)
I20260925 15:07:20.133893 8525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:37033
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:20.162716 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:20.472122 8721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:20.472460 8721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:20.472617 8721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:20.479624 8721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:20.479741 8721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:20.479923 8721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:20.488624 8721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:20.489799 8721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:20.490901 8721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:20.502600 8726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.502561 8729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.502683 8727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:20.503575 8721 server_base.cc:1034] running on GCE node
I20260925 15:07:20.504160 8721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:20.504941 8721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:20.506141 8721 hybrid_clock.cc:648] HybridClock initialized: now 1790348840506090 us; error 49 us; skew 500 ppm
I20260925 15:07:20.506425 8721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:20 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:20.509275 8721 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:20.510614 8721 webserver.cc:460] Webserver started at http://127.4.205.194:35437/ using document root <none> and password file <none>
I20260925 15:07:20.511248 8721 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:20.511375 8721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:20.511675 8721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:20.514323 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "31a9f58fb0304b4093ec1db4ab8056cf"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "c5142be8c5c29f4a431966dd007a4f54"
server_key_iv: "719f0659d60d75cc093ba36dce4c610a"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.515029 8721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "31a9f58fb0304b4093ec1db4ab8056cf"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "c5142be8c5c29f4a431966dd007a4f54"
server_key_iv: "719f0659d60d75cc093ba36dce4c610a"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.519518 8721 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:20.522655 8736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.523808 8721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:20.523974 8721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "31a9f58fb0304b4093ec1db4ab8056cf"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "c5142be8c5c29f4a431966dd007a4f54"
server_key_iv: "719f0659d60d75cc093ba36dce4c610a"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.524156 8721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:20.544154 8721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:20.547614 8721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:20.547887 8721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:20.548709 8721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:20.549953 8721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:20.550050 8721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.550190 8721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:20.550261 8721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.581672 8721 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:41297
I20260925 15:07:20.581709 8848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:41297 every 8 connection(s)
I20260925 15:07:20.582975 8721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:20.590528 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8721
I20260925 15:07:20.590718 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:20.591176 4919 external_mini_cluster.cc:1428] Setting key ef3e01c2efe8b56069334cf72a50657e
Sep 25 15:07:20 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:20.602572 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.585352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44335 (local address 127.4.205.254:42971)
0925 15:07:20.585581 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:20.585589 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:20.586681 (+ 1092us) server_negotiation.cc:407] Connection header received
0925 15:07:20.588328 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:20.588332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:20.588391 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:20.588477 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:20.589843 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.590819 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.592427 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.593034 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.593130 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.597059 (+ 3929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:20.597064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:20.597067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:20.597095 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:20.599704 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.600373 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.600379 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.600385 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.600507 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:20.600918 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:20.600924 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:20.600928 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:20.601169 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:20.601353 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:20.602046 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 15:07:20.602250 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:07:20.603478 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:20.585855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42971 (local address 127.4.205.194:44335)
0925 15:07:20.586527 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:20.586561 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:20.588091 (+ 1530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:20.588655 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:20.588663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:20.589111 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:20.589705 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.589727 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.590955 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.590958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.592300 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:20.592311 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:20.593237 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:20.593244 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:20.593515 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:20.594230 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:20.594262 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:20.596847 (+ 2585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:20.599826 (+ 2979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.599835 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.599849 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.600231 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.600613 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:20.600619 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:20.600624 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:20.600795 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:20.601318 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:20.601326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:20.601704 (+ 378us) client_negotiation.cc:769] Sending connection context
0925 15:07:20.601952 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:07:20.602420 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":265,"threads_started":1}
I20260925 15:07:20.604807 8849 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42971
I20260925 15:07:20.605101 8849 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:20.605795 8849 heartbeater.cc:511] Master 127.4.205.254:42971 requested a full tablet report, sending...
I20260925 15:07:20.607185 8525 ts_manager.cc:194] Registered new tserver with Master: 31a9f58fb0304b4093ec1db4ab8056cf (127.4.205.194:41297)
I20260925 15:07:20.608359 8525 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:44335
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:20.633358 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:38677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:20.936236 8854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:20.936491 8854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:20.936580 8854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:20.943553 8854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:20.943670 8854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:20.943779 8854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:20.952833 8854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:38677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42971
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:20.953990 8854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:20.955161 8854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:20.968219 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.968264 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:20.968312 8862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:20.969019 8854 server_base.cc:1034] running on GCE node
I20260925 15:07:20.969697 8854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:20.970635 8854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:20.971849 8854 hybrid_clock.cc:648] HybridClock initialized: now 1790348840971802 us; error 62 us; skew 500 ppm
I20260925 15:07:20.972153 8854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:20 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:20.975142 8854 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:20.976508 8854 webserver.cc:460] Webserver started at http://127.4.205.195:35491/ using document root <none> and password file <none>
I20260925 15:07:20.977188 8854 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:20.977329 8854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:20.977689 8854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:20.980504 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "a88b2f9071ae460f8a2847a7a910348c"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "76ce8fb658e4622ae0cf96aabfa6c317"
server_key_iv: "51eadb2df5adb1d1dbe3e6501ead371c"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.981258 8854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "a88b2f9071ae460f8a2847a7a910348c"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "76ce8fb658e4622ae0cf96aabfa6c317"
server_key_iv: "51eadb2df5adb1d1dbe3e6501ead371c"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.986132 8854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:07:20.989770 8869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:20.990933 8854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:20.991102 8854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "a88b2f9071ae460f8a2847a7a910348c"
format_stamp: "Formatted at 2026-09-25 15:07:20 on dist-test-slave-64fx"
server_key: "76ce8fb658e4622ae0cf96aabfa6c317"
server_key_iv: "51eadb2df5adb1d1dbe3e6501ead371c"
server_key_version: "encryptionkey@0"
I20260925 15:07:20.991308 8854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:21.012029 8854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:21.015748 8854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:21.016021 8854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:21.016974 8854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:21.018308 8854 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:21.018412 8854 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:21.018559 8854 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:21.018630 8854 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:21.050900 8854 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:43363
I20260925 15:07:21.050979 8981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:43363 every 8 connection(s)
I20260925 15:07:21.052171 8854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:21.060490 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 8854
I20260925 15:07:21.060688 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:21.061084 4919 external_mini_cluster.cc:1428] Setting key 5ce4a59c72ce4800cae5bc80958ce93d
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348840, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:21.071241 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.054672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:36503 (local address 127.4.205.254:42971)
0925 15:07:21.054878 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.054883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.055898 (+ 1015us) server_negotiation.cc:407] Connection header received
0925 15:07:21.057810 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.057816 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.057917 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.058029 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.059507 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.060256 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.062122 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.062647 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.062791 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.066256 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.066261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.066265 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.066296 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.068671 (+ 2375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.069280 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.069284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.069287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.069394 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.069829 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.069833 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.069835 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.070070 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.070170 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.070834 (+ 664us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.070998 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:07:21.072187 8984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.055075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42971 (local address 127.4.205.195:36503)
0925 15:07:21.055743 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:21.055779 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:21.057462 (+ 1683us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:21.058287 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:21.058296 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:21.058800 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:21.059345 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:21.059362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.060412 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.060419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:21.061979 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:21.061991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.062873 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.062882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:21.063130 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.063838 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:21.063883 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:21.066063 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:21.068788 (+ 2725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:21.068794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:21.068805 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:21.069177 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:21.069479 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:21.069482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:21.069484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:21.069736 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:21.070172 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:21.070177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:21.070531 (+ 354us) client_negotiation.cc:769] Sending connection context
0925 15:07:21.070747 (+ 216us) client_negotiation.cc:240] Negotiation successful
0925 15:07:21.071154 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":258,"threads_started":1}
I20260925 15:07:21.073634 8982 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42971
I20260925 15:07:21.073937 8982 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:21.074733 8982 heartbeater.cc:511] Master 127.4.205.254:42971 requested a full tablet report, sending...
I20260925 15:07:21.076323 8524 ts_manager.cc:194] Registered new tserver with Master: a88b2f9071ae460f8a2847a7a910348c (127.4.205.195:43363)
I20260925 15:07:21.077126 8524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:36503
I20260925 15:07:21.088145 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:21.135226 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.121762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57864 (local address 127.4.205.193:40191)
0925 15:07:21.122460 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.122467 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.122492 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:21.122547 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.122553 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.122779 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.122899 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.124152 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.125120 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.126317 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.126853 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.126948 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.130677 (+ 3729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.130687 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.130699 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.130731 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.133129 (+ 2398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.133701 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.133708 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.133710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.133820 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.134109 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.134112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.134114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.134410 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.134612 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.134781 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.134952 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":404,"threads_started":1}
I20260925 15:07:21.137132 8715 heartbeater.cc:503] Master 127.4.205.254:42971 was elected leader, sending a full tablet report...
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:21.151419 8717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.139734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.4.205.254:42971)
0925 15:07:21.139878 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.139897 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.140247 (+ 350us) server_negotiation.cc:407] Connection header received
0925 15:07:21.140354 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.140357 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.140411 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.140481 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.141532 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.142291 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.143408 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.143799 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.143884 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.147252 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.147257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.147260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.147295 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.149411 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.149980 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.149983 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.149986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.150101 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.150393 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.150396 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.150398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.150665 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.150828 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.151060 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.151201 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 15:07:21.151926 8524 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40732
I20260925 15:07:21.152846 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8587
I20260925 15:07:21.164413 8989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.153766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.4.205.193:40191)
0925 15:07:21.153956 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.153962 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.154071 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:07:21.154182 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.154186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.154240 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.154316 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.155357 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.156011 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.157105 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.157493 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.157643 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.159713 (+ 2070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.159717 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.159721 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.159745 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.162354 (+ 2609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.162928 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.162932 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.162934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.163042 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.163334 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.163337 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.163338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.163554 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.163709 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.163942 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.164087 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:21.165066 8710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:21.246277 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8587
I20260925 15:07:21.263365 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8721
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:21.278329 8994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.264191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55640 (local address 127.4.205.194:41297)
0925 15:07:21.264786 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.264793 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.264816 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:21.264874 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.264880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.265094 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.265221 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.266403 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.267451 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.268554 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.269063 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.269151 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.273120 (+ 3969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.273135 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.273149 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.273186 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.276048 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.276618 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.276625 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.276627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.276737 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.277027 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.277030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.277032 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.277334 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.277545 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.277824 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.278016 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":351,"threads_started":1}
I20260925 15:07:21.279454 8844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:21.358035 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8721
I20260925 15:07:21.373108 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8854
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[8480](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348841, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:21.388263 8996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:21.374111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58456 (local address 127.4.205.195:43363)
0925 15:07:21.374706 (+ 595us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:21.374713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:21.374738 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:21.374794 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:21.374800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:21.375018 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:21.375185 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:21.376381 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.377414 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.378751 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:21.379288 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:21.379426 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:21.383440 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:21.383450 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:21.383462 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:21.383495 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:21.386093 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.386649 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.386655 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.386657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.386762 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:21.387088 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:21.387091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:21.387093 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:21.387392 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:21.387610 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:21.387815 (+ 205us) server_negotiation.cc:299] Negotiation successful
0925 15:07:21.387990 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":309,"threads_started":1}
I20260925 15:07:21.389436 8977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:21.468237 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8854
I20260925 15:07:21.483263 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 8493
I20260925 15:07:21.484406 8555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:21.583146 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 8493
2026-09-25T15:07:21Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3693 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[9001](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[9001](info): set up 2 sockets
Sep 25 15:07:21 dist-test-slave-64fx krb5kdc[9001](info): commencing operation
krb5kdc: starting...
W20260925 15:07:22.637953 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:22 dist-test-slave-64fx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:22.821445 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39167
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39167
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:23.121965 9014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:23.122256 9014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:23.122427 9014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:23.129380 9014 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:23.129504 9014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:23.129629 9014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:23.129724 9014 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:23.129820 9014 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:23.139091 9014 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39167
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:23.140298 9014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:23.141626 9014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:23.152961 9019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.152912 9022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.152994 9020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:23.153748 9014 server_base.cc:1034] running on GCE node
I20260925 15:07:23.154448 9014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:23.155794 9014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:23.156985 9014 hybrid_clock.cc:648] HybridClock initialized: now 1790348843156946 us; error 29 us; skew 500 ppm
I20260925 15:07:23.157323 9014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:23 dist-test-slave-64fx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:23.160234 9014 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:23.161252 9014 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:23.161342 9014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:23.161640 9014 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:23.164343 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "994bb218d60745708253536b6c8ac19f"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "fe5f923d3be6e652400fc2434159bdfb"
server_key_iv: "d07906e6b8edb5709bdca5ef9b2312a0"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.165019 9014 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "994bb218d60745708253536b6c8ac19f"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "fe5f923d3be6e652400fc2434159bdfb"
server_key_iv: "d07906e6b8edb5709bdca5ef9b2312a0"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.169713 9014 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:07:23.172789 9028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:23.174062 9014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:23.174229 9014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "994bb218d60745708253536b6c8ac19f"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "fe5f923d3be6e652400fc2434159bdfb"
server_key_iv: "d07906e6b8edb5709bdca5ef9b2312a0"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.174407 9014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:23.194633 9014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:23.198195 9014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:23.198443 9014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:23.215947 9014 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:39167
I20260925 15:07:23.215970 9079 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:39167 every 8 connection(s)
I20260925 15:07:23.216784 9014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:23.220130 9080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:23.226200 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9014
I20260925 15:07:23.226462 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:23.227097 4919 external_mini_cluster.cc:1428] Setting key d475b81711cccc786a25e8696b7397d1
I20260925 15:07:23.231199 9080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: Bootstrap starting.
I20260925 15:07:23.234709 9080 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:23.236055 9080 log.cc:824] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:23.238576 9080 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: No bootstrap required, opened a new log
Sep 25 15:07:23 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:23.245990 9080 raft_consensus.cc:348] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } }
I20260925 15:07:23.246344 9080 raft_consensus.cc:374] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:23.246452 9080 raft_consensus.cc:729] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 994bb218d60745708253536b6c8ac19f, State: Initialized, Role: FOLLOWER
I20260925 15:07:23.247087 9080 consensus_queue.cc:260] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } }
I20260925 15:07:23.247382 9080 raft_consensus.cc:388] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:23.247534 9080 raft_consensus.cc:482] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:23.247710 9080 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:23.250346 9083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:23.229431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48760 (local address 127.4.205.254:39167)
0925 15:07:23.230192 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:23.230207 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:23.230256 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:07:23.232544 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:23.232569 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:23.233076 (+ 507us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:23.233489 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:23.234856 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.236878 (+ 2022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.238206 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.238938 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.239135 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:23.242850 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:23.242858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:23.242874 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:23.242908 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:23.245844 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.246574 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.246583 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.246590 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.246794 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.247172 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.247177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.247180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.247894 (+ 714us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:23.248183 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:23.248530 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 15:07:23.248963 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260925 15:07:23.250905 9080 raft_consensus.cc:504] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } }
I20260925 15:07:23.251360 9080 leader_election.cc:302] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 994bb218d60745708253536b6c8ac19f; no voters:
I20260925 15:07:23.252002 9080 leader_election.cc:288] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:23.252142 9085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:23.253966 9085 raft_consensus.cc:686] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [term 1 LEADER]: Becoming Leader. State: Replica: 994bb218d60745708253536b6c8ac19f, State: Running, Role: LEADER
I20260925 15:07:23.254163 9080 sys_catalog.cc:564] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:23.254464 9085 consensus_queue.cc:237] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } }
I20260925 15:07:23.257426 9086 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "994bb218d60745708253536b6c8ac19f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } } }
I20260925 15:07:23.257407 9087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 994bb218d60745708253536b6c8ac19f. Latest consensus state: current_term: 1 leader_uuid: "994bb218d60745708253536b6c8ac19f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "994bb218d60745708253536b6c8ac19f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39167 } } }
I20260925 15:07:23.257865 9086 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:23.257922 9087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:23.258706 9095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:23.266780 9095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:23.275303 9095 catalog_manager.cc:1347] Generated new cluster ID: 58b7547766f54476ba83e9feb5126059
I20260925 15:07:23.275420 9095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:23.288064 9095 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:23.289445 9095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:23.299309 9095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: Generated new TSK 0
I20260925 15:07:23.300122 9095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:23.357241 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:23.670104 9106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:23.670362 9106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:23.670547 9106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:23.677358 9106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:23.677456 9106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:23.677680 9106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:23.687034 9106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:23.688213 9106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:23.689469 9106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:23.701656 9112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.701718 9111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:23.701797 9114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:23.702342 9106 server_base.cc:1034] running on GCE node
I20260925 15:07:23.702878 9106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:23.703711 9106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:23.704928 9106 hybrid_clock.cc:648] HybridClock initialized: now 1790348843704886 us; error 37 us; skew 500 ppm
I20260925 15:07:23.705233 9106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:23 dist-test-slave-64fx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:23.708250 9106 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:23.709225 9106 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:23.709306 9106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:23.709533 9106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:23.712191 9106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "2adffe60e98a47ec914a34c5aa5af33e"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "b97c78bcb9e5890098ca55876e3fdc1b"
server_key_iv: "93e2b11fbc3a44964aa11cad6d23535d"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.712898 9106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "2adffe60e98a47ec914a34c5aa5af33e"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "b97c78bcb9e5890098ca55876e3fdc1b"
server_key_iv: "93e2b11fbc3a44964aa11cad6d23535d"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.717274 9106 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.003s
I20260925 15:07:23.720317 9120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:23.721490 9106 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:23.721728 9106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "2adffe60e98a47ec914a34c5aa5af33e"
format_stamp: "Formatted at 2026-09-25 15:07:23 on dist-test-slave-64fx"
server_key: "b97c78bcb9e5890098ca55876e3fdc1b"
server_key_iv: "93e2b11fbc3a44964aa11cad6d23535d"
server_key_version: "encryptionkey@0"
I20260925 15:07:23.721923 9106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:23.739796 9106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:23.743420 9106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:23.743690 9106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:23.744560 9106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:23.745919 9106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:23.746027 9106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:23.746155 9106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:23.746251 9106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:23.778250 9106 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:42675
I20260925 15:07:23.778321 9232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:42675 every 8 connection(s)
I20260925 15:07:23.779084 9106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:23.786598 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9106
I20260925 15:07:23.786814 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:23.787629 4919 external_mini_cluster.cc:1428] Setting key 9356529693cfa32ab2e07fad4415f631
Sep 25 15:07:23 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348843, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:23.800307 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:23.781479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:32957 (local address 127.4.205.254:39167)
0925 15:07:23.781915 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:23.781920 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:23.782923 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 15:07:23.784537 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:23.784541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:23.784608 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:23.784698 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:23.786119 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.786921 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.789016 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.789625 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.789731 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:23.793748 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:23.793753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:23.793756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:23.793783 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:23.796781 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.797715 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.797719 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.797721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.797832 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:23.798446 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:23.798450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:23.798452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:23.798690 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:23.798859 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:23.799576 (+ 717us) server_negotiation.cc:299] Negotiation successful
0925 15:07:23.799818 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":251,"threads_started":1}
I20260925 15:07:23.800782 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:23.782104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39167 (local address 127.4.205.193:32957)
0925 15:07:23.782773 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:23.782811 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:23.784308 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:23.784889 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:23.784896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:23.785352 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:23.785974 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:23.785991 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.787081 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.787087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:23.788875 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:23.788886 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:23.789844 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:23.789850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:23.790109 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:23.790961 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:23.790997 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:23.793487 (+ 2490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:23.796916 (+ 3429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:23.796927 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:23.796945 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:23.797597 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:23.797934 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:23.797942 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:23.797948 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:23.798175 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:23.798841 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:23.798849 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:23.799143 (+ 294us) client_negotiation.cc:769] Sending connection context
0925 15:07:23.799416 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 15:07:23.799799 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":270,"threads_started":1}
I20260925 15:07:23.802311 9233 heartbeater.cc:347] Connected to a master server at 127.4.205.254:39167
I20260925 15:07:23.802619 9233 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:23.803395 9233 heartbeater.cc:511] Master 127.4.205.254:39167 requested a full tablet report, sending...
I20260925 15:07:23.805406 9045 ts_manager.cc:194] Registered new tserver with Master: 2adffe60e98a47ec914a34c5aa5af33e (127.4.205.193:42675)
I20260925 15:07:23.806797 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:32957
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:23.832599 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:24.137130 9239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:24.137485 9239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:24.137707 9239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:24.144479 9239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:24.144604 9239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:24.144798 9239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:24.153682 9239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:24.154956 9239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:24.156162 9239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:24.168712 9244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:24.168742 9245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:24.168717 9247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:24.169785 9239 server_base.cc:1034] running on GCE node
I20260925 15:07:24.170420 9239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:24.171319 9239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:24.172545 9239 hybrid_clock.cc:648] HybridClock initialized: now 1790348844172502 us; error 55 us; skew 500 ppm
I20260925 15:07:24.172860 9239 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:24 dist-test-slave-64fx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:24.176038 9239 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:24.177161 9239 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:24.177295 9239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:24.177704 9239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:24.180480 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "6d2eac08a2564ad99c2f51c0ac52c10c"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "034c0a35102f59da1bb8b8485abee56e"
server_key_iv: "aa66482109f865a273fced9e25b111db"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.181272 9239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "6d2eac08a2564ad99c2f51c0ac52c10c"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "034c0a35102f59da1bb8b8485abee56e"
server_key_iv: "aa66482109f865a273fced9e25b111db"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.187161 9239 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:07:24.190294 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.191584 9239 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:24.191752 9239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "6d2eac08a2564ad99c2f51c0ac52c10c"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "034c0a35102f59da1bb8b8485abee56e"
server_key_iv: "aa66482109f865a273fced9e25b111db"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.191951 9239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:24.209662 9239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:24.213375 9239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:24.213657 9239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:24.214586 9239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:24.216007 9239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:24.216100 9239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.216249 9239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:24.216325 9239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.249280 9239 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:33069
I20260925 15:07:24.249333 9365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:33069 every 8 connection(s)
I20260925 15:07:24.250128 9239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:24.259251 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9239
I20260925 15:07:24.259434 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:24.259892 4919 external_mini_cluster.cc:1428] Setting key 2966201f3a0573f0319292627094cf44
Sep 25 15:07:24 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:24.269277 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.252480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:43041 (local address 127.4.205.254:39167)
0925 15:07:24.252662 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.252667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.253856 (+ 1189us) server_negotiation.cc:407] Connection header received
0925 15:07:24.255797 (+ 1941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.255801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.255867 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.255953 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:24.257539 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.258356 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.259920 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.260478 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.260610 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.263908 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:24.263913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:24.263916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:24.263943 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:24.266351 (+ 2408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.267051 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.267059 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.267063 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.267188 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.267591 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.267616 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.267618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.267927 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:24.268076 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.268735 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.268949 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:07:24.270329 9368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.252981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39167 (local address 127.4.205.194:43041)
0925 15:07:24.253701 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.253740 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.255509 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.256148 (+ 639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.256156 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.256725 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:24.257346 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.257369 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.258474 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.258477 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.259790 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.259801 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.260691 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.260698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.260913 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.261667 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:24.261688 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:24.263684 (+ 1996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:24.266504 (+ 2820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:24.266514 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:24.266532 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:24.266936 (+ 404us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:24.267294 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:24.267301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:24.267307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:24.267483 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:24.268050 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:24.268057 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:24.268399 (+ 342us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.268665 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.269185 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":253,"threads_started":1}
I20260925 15:07:24.271714 9366 heartbeater.cc:347] Connected to a master server at 127.4.205.254:39167
I20260925 15:07:24.272054 9366 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:24.272817 9366 heartbeater.cc:511] Master 127.4.205.254:39167 requested a full tablet report, sending...
I20260925 15:07:24.274358 9045 ts_manager.cc:194] Registered new tserver with Master: 6d2eac08a2564ad99c2f51c0ac52c10c (127.4.205.194:33069)
I20260925 15:07:24.275148 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:43041
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:24.304476 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:24.615890 9371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:24.616123 9371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:24.616220 9371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:24.623023 9371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:24.623104 9371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:24.623207 9371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:24.632099 9371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:39167
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:24.633136 9371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:24.634268 9371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:24.646279 9379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:24.646380 9376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:24.646373 9377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:24.646960 9371 server_base.cc:1034] running on GCE node
I20260925 15:07:24.647626 9371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:24.648417 9371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:24.649657 9371 hybrid_clock.cc:648] HybridClock initialized: now 1790348844649619 us; error 43 us; skew 500 ppm
I20260925 15:07:24.649957 9371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:24 dist-test-slave-64fx krb5kdc[9001](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:24.656623 9371 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:24.657824 9371 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:24.657936 9371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:24.658221 9371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:24.660845 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "b942b1777b904e318fa4fe8b9979bd3f"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "1620125e6de2c55262b5ca772089865e"
server_key_iv: "c381fd53130d7a446ba38fcd13050c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.661711 9371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "b942b1777b904e318fa4fe8b9979bd3f"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "1620125e6de2c55262b5ca772089865e"
server_key_iv: "c381fd53130d7a446ba38fcd13050c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.666376 9371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:07:24.669697 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.671129 9371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:24.671309 9371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "b942b1777b904e318fa4fe8b9979bd3f"
format_stamp: "Formatted at 2026-09-25 15:07:24 on dist-test-slave-64fx"
server_key: "1620125e6de2c55262b5ca772089865e"
server_key_iv: "c381fd53130d7a446ba38fcd13050c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:24.671519 9371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:24.702557 9371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:24.706655 9371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:24.706921 9371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:24.707820 9371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:24.709019 9371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:24.709134 9371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.709254 9371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:24.709347 9371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:24.741109 9371 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:35131
I20260925 15:07:24.741199 9497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:35131 every 8 connection(s)
I20260925 15:07:24.741992 9371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:24.751479 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9371
I20260925 15:07:24.751662 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:24.752180 4919 external_mini_cluster.cc:1428] Setting key 3c0a387447c8ef78489fe05d0aa3ac74
Sep 25 15:07:24 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348844, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:24.760279 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.744356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:39295 (local address 127.4.205.254:39167)
0925 15:07:24.744524 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.744529 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.745604 (+ 1075us) server_negotiation.cc:407] Connection header received
0925 15:07:24.747301 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.747305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.747371 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.747456 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:24.748778 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.749469 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.750923 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.751285 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.751376 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.755238 (+ 3862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:24.755243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:24.755246 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:24.755273 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:24.757973 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.758573 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.758576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.758578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.758677 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.759018 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.759021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.759023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.759245 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:24.759363 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.759927 (+ 564us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.760062 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:24.761098 9500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.744766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39167 (local address 127.4.205.195:39295)
0925 15:07:24.745409 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.745450 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.747057 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.747638 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.747645 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.748123 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:24.748627 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.748647 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.749629 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.749641 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.750803 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.750813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.751511 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.751519 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.751793 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.752470 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:24.752490 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:24.755072 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:24.758073 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:24.758079 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:24.758090 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:24.758477 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:24.758755 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:24.758758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:24.758760 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:24.758928 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:24.759356 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:24.759361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:24.759634 (+ 273us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.759839 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.760159 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":234,"threads_started":1}
I20260925 15:07:24.762399 9498 heartbeater.cc:347] Connected to a master server at 127.4.205.254:39167
I20260925 15:07:24.762642 9498 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:24.763162 9498 heartbeater.cc:511] Master 127.4.205.254:39167 requested a full tablet report, sending...
I20260925 15:07:24.764310 9045 ts_manager.cc:194] Registered new tserver with Master: b942b1777b904e318fa4fe8b9979bd3f (127.4.205.195:35131)
I20260925 15:07:24.764963 9045 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:39295
I20260925 15:07:24.767000 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:24.783017 9235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.771122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48766 (local address 127.4.205.254:39167)
0925 15:07:24.771321 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.771329 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.771809 (+ 480us) server_negotiation.cc:407] Connection header received
0925 15:07:24.771991 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.771996 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.772069 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.772160 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:24.773436 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.774311 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.775554 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.776046 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.776166 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.778264 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:24.778269 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:24.778274 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:24.778308 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:24.780511 (+ 2203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.781132 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.781138 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.781141 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.781291 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:24.781684 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:24.781690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:24.781695 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:24.781979 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:24.782211 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.782584 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.782759 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:07:24.786211 9045 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:24.792292 9045 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:24.809589 9233 heartbeater.cc:503] Master 127.4.205.254:39167 was elected leader, sending a full tablet report...
I20260925 15:07:24.812649 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.804114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46752 (local address 127.4.205.193:42675)
0925 15:07:24.805465 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.805477 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.805506 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:07:24.805628 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.805634 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.805903 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.806109 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:24.806985 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.808175 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.810709 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.811631 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.811771 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.811975 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.812203 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.812356 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1158,"thread_start_us":1017,"threads_started":1}
I20260925 15:07:24.813040 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.802942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:33069 (local address 127.0.0.1:35124)
0925 15:07:24.803701 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.803738 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.803886 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.805812 (+ 1926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.805819 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.805851 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:24.806348 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.806366 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.808221 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.808227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.810594 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.810607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.812292 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.812298 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.812456 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.812485 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.812548 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.812674 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"mutex_wait_us":71,"thread_start_us":381,"threads_started":1}
I20260925 15:07:24.813372 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.803541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:42675 (local address 127.0.0.1:46752)
0925 15:07:24.804476 (+ 935us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.804496 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.804613 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.806349 (+ 1736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.806357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.806386 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:24.806786 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.806801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.808304 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.808308 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.810550 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.810564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.811829 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.811835 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.811998 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.812051 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.812168 (+ 117us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.812336 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":27,"thread_start_us":782,"threads_started":1}
I20260925 15:07:24.814173 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.803248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.4.205.194:33069)
0925 15:07:24.805138 (+ 1890us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.805146 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.805191 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:07:24.805294 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.805300 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.805566 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.806540 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:24.806742 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.808091 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.810778 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.812082 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.813437 (+ 1355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.813632 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.813697 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.813851 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":1563,"threads_started":1}
I20260925 15:07:24.814373 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.804192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:35131 (local address 127.0.0.1:44392)
0925 15:07:24.806276 (+ 2084us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.806295 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.806413 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.807022 (+ 609us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.807026 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.807054 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:24.807405 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.807416 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.809166 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.809171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.811691 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.811704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.813689 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.813696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.813858 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.813886 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.813962 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.814074 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1931,"thread_start_us":1775,"threads_started":1}
I20260925 15:07:24.815291 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.804857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44392 (local address 127.4.205.195:35131)
0925 15:07:24.805816 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.805824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.806532 (+ 708us) server_negotiation.cc:407] Connection header received
0925 15:07:24.806629 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.806635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.806842 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.807027 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:24.807614 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.809011 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.811866 (+ 2855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.813472 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.814558 (+ 1086us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.814756 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.814814 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.814954 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":619,"threads_started":1}
I20260925 15:07:24.816887 9168 tablet_service.cc:1467] Processing CreateTablet for tablet bb9c8add76104eca8ece54b3ca57fcef (DEFAULT_TABLE table=test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:24.817458 9301 tablet_service.cc:1467] Processing CreateTablet for tablet bb9c8add76104eca8ece54b3ca57fcef (DEFAULT_TABLE table=test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:24.818312 9168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9c8add76104eca8ece54b3ca57fcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:24.818259 9433 tablet_service.cc:1467] Processing CreateTablet for tablet bb9c8add76104eca8ece54b3ca57fcef (DEFAULT_TABLE table=test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:24.818710 9301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9c8add76104eca8ece54b3ca57fcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:24.819247 9433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9c8add76104eca8ece54b3ca57fcef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:24.829758 9516 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Bootstrap starting.
I20260925 15:07:24.831840 9518 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Bootstrap starting.
I20260925 15:07:24.832496 9517 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Bootstrap starting.
I20260925 15:07:24.833187 9516 tablet_bootstrap.cc:654] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:24.834671 9516 log.cc:824] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:24.835492 9518 tablet_bootstrap.cc:654] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:24.835636 9517 tablet_bootstrap.cc:654] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:24.836486 9518 log.cc:824] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:24.836750 9517 log.cc:824] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:24.837833 9516 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: No bootstrap required, opened a new log
I20260925 15:07:24.838115 9516 ts_tablet_manager.cc:1397] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:07:24.839690 9517 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: No bootstrap required, opened a new log
I20260925 15:07:24.839690 9518 tablet_bootstrap.cc:492] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: No bootstrap required, opened a new log
I20260925 15:07:24.839928 9517 ts_tablet_manager.cc:1397] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:07:24.839927 9518 ts_tablet_manager.cc:1397] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:07:24.845992 9516 raft_consensus.cc:348] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.846297 9516 raft_consensus.cc:374] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:24.846421 9516 raft_consensus.cc:729] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b942b1777b904e318fa4fe8b9979bd3f, State: Initialized, Role: FOLLOWER
I20260925 15:07:24.847002 9516 consensus_queue.cc:260] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.847982 9498 heartbeater.cc:503] Master 127.4.205.254:39167 was elected leader, sending a full tablet report...
I20260925 15:07:24.848580 9516 ts_tablet_manager.cc:1428] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20260925 15:07:24.849889 9517 raft_consensus.cc:348] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.849889 9518 raft_consensus.cc:348] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.850256 9517 raft_consensus.cc:374] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:24.850253 9518 raft_consensus.cc:374] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:24.850350 9518 raft_consensus.cc:729] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d2eac08a2564ad99c2f51c0ac52c10c, State: Initialized, Role: FOLLOWER
I20260925 15:07:24.850350 9517 raft_consensus.cc:729] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2adffe60e98a47ec914a34c5aa5af33e, State: Initialized, Role: FOLLOWER
I20260925 15:07:24.850870 9518 consensus_queue.cc:260] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.850871 9517 consensus_queue.cc:260] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.852651 9517 ts_tablet_manager.cc:1428] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Time spent starting tablet: real 0.013s user 0.004s sys 0.009s
I20260925 15:07:24.854229 9366 heartbeater.cc:503] Master 127.4.205.254:39167 was elected leader, sending a full tablet report...
I20260925 15:07:24.854857 9518 ts_tablet_manager.cc:1428] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260925 15:07:24.894675 9524 raft_consensus.cc:482] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:24.894940 9524 raft_consensus.cc:504] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.896677 9524 leader_election.cc:288] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d2eac08a2564ad99c2f51c0ac52c10c (127.4.205.194:33069), b942b1777b904e318fa4fe8b9979bd3f (127.4.205.195:35131)
I20260925 15:07:24.903762 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.897017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:58057 (local address 127.4.205.194:33069)
0925 15:07:24.897235 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.897244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.897708 (+ 464us) server_negotiation.cc:407] Connection header received
0925 15:07:24.897954 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.897960 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.898038 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.898099 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:24.898763 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.899708 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.902007 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.902965 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.903124 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.903206 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.903408 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.903526 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:24.903788 9525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.896982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:33069 (local address 127.4.205.193:58057)
0925 15:07:24.897596 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.897622 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.897776 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.898237 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.898239 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.898257 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:24.898634 (+ 377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.898641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.899820 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.899823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.901869 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.901883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.903152 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.903156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.903278 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.903299 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.903366 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.903451 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":23,"thread_start_us":330,"threads_started":1}
I20260925 15:07:24.904496 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.897964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:47105 (local address 127.4.205.195:35131)
0925 15:07:24.898197 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.898204 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.898796 (+ 592us) server_negotiation.cc:407] Connection header received
0925 15:07:24.898941 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.898945 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.899010 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.899101 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:24.899516 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.900271 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.902471 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.903544 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.903678 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.903732 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.904148 (+ 416us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.904262 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 15:07:24.904567 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.897779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:35131 (local address 127.4.205.193:47105)
0925 15:07:24.898718 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:24.898731 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:24.898820 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:24.899117 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:24.899120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:24.899136 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:24.899375 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.899381 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.900397 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.900402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.902315 (+ 1913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:24.902325 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.903755 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.903762 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:24.903953 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.904003 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:07:24.904123 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 15:07:24.904235 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":3,"thread_start_us":664,"threads_started":1}
I20260925 15:07:24.904675 9321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9c8add76104eca8ece54b3ca57fcef" candidate_uuid: "2adffe60e98a47ec914a34c5aa5af33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" is_pre_election: true
I20260925 15:07:24.905191 9321 raft_consensus.cc:2454] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adffe60e98a47ec914a34c5aa5af33e in term 0.
I20260925 15:07:24.905435 9453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9c8add76104eca8ece54b3ca57fcef" candidate_uuid: "2adffe60e98a47ec914a34c5aa5af33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b942b1777b904e318fa4fe8b9979bd3f" is_pre_election: true
I20260925 15:07:24.905875 9121 leader_election.cc:302] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adffe60e98a47ec914a34c5aa5af33e, 6d2eac08a2564ad99c2f51c0ac52c10c; no voters:
I20260925 15:07:24.905926 9453 raft_consensus.cc:2454] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2adffe60e98a47ec914a34c5aa5af33e in term 0.
I20260925 15:07:24.906217 9524 raft_consensus.cc:2781] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:24.906399 9524 raft_consensus.cc:482] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:24.906546 9524 raft_consensus.cc:3037] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:24.908615 9524 raft_consensus.cc:504] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.909114 9524 leader_election.cc:288] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [CANDIDATE]: Term 1 election: Requested vote from peers 6d2eac08a2564ad99c2f51c0ac52c10c (127.4.205.194:33069), b942b1777b904e318fa4fe8b9979bd3f (127.4.205.195:35131)
I20260925 15:07:24.909533 9321 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9c8add76104eca8ece54b3ca57fcef" candidate_uuid: "2adffe60e98a47ec914a34c5aa5af33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c"
I20260925 15:07:24.909664 9453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9c8add76104eca8ece54b3ca57fcef" candidate_uuid: "2adffe60e98a47ec914a34c5aa5af33e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b942b1777b904e318fa4fe8b9979bd3f"
I20260925 15:07:24.909771 9321 raft_consensus.cc:3037] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:24.909881 9453 raft_consensus.cc:3037] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:24.911855 9321 raft_consensus.cc:2454] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adffe60e98a47ec914a34c5aa5af33e in term 1.
I20260925 15:07:24.912029 9453 raft_consensus.cc:2454] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2adffe60e98a47ec914a34c5aa5af33e in term 1.
I20260925 15:07:24.912196 9121 leader_election.cc:302] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2adffe60e98a47ec914a34c5aa5af33e, 6d2eac08a2564ad99c2f51c0ac52c10c; no voters:
I20260925 15:07:24.912468 9524 raft_consensus.cc:2781] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:24.912869 9524 raft_consensus.cc:686] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 1 LEADER]: Becoming Leader. State: Replica: 2adffe60e98a47ec914a34c5aa5af33e, State: Running, Role: LEADER
I20260925 15:07:24.913326 9524 consensus_queue.cc:237] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } }
I20260925 15:07:24.917346 9044 catalog_manager.cc:5495] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e reported cstate change: term changed from 0 to 1, leader changed from <none> to 2adffe60e98a47ec914a34c5aa5af33e (127.4.205.193). New cstate: current_term: 1 leader_uuid: "2adffe60e98a47ec914a34c5aa5af33e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2adffe60e98a47ec914a34c5aa5af33e" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42675 } health_report { overall_health: HEALTHY } } }
I20260925 15:07:24.936219 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.931254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.4.205.193:42675)
0925 15:07:24.931436 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.931443 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.931554 (+ 111us) server_negotiation.cc:407] Connection header received
0925 15:07:24.931774 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.931777 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.931850 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.931997 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:24.932355 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.933027 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.934304 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.934666 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.934788 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:24.935199 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:24.935638 (+ 439us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:24.935801 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:24.935909 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:07:24.935983 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:07:24.943988 9321 raft_consensus.cc:1261] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 1 FOLLOWER]: Refusing update from remote peer 2adffe60e98a47ec914a34c5aa5af33e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:24.944027 9453 raft_consensus.cc:1261] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 1 FOLLOWER]: Refusing update from remote peer 2adffe60e98a47ec914a34c5aa5af33e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:24.944913 9527 consensus_queue.cc:1035] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d2eac08a2564ad99c2f51c0ac52c10c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:24.945247 9524 consensus_queue.cc:1035] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b942b1777b904e318fa4fe8b9979bd3f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:24.958233 9532 mvcc.cc:205] Tried to move back new op lower bound from 7333268868880498688 to 7333268868768608256. Current Snapshot: MvccSnapshot[applied={T|T < 7333268868880498688}]
I20260925 15:07:24.968608 9044 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:24.968897 9044 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48766:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:24.972095 9044 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 994bb218d60745708253536b6c8ac19f: Sending DeleteTablet for 3 replicas of tablet bb9c8add76104eca8ece54b3ca57fcef
I20260925 15:07:24.973093 9301 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9c8add76104eca8ece54b3ca57fcef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:24 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:35124
I20260925 15:07:24.973129 9433 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9c8add76104eca8ece54b3ca57fcef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:24 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:44392
I20260925 15:07:24.973707 9168 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9c8add76104eca8ece54b3ca57fcef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:24 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:46752
I20260925 15:07:24.973918 9540 tablet_replica.cc:330] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: stopping tablet replica
I20260925 15:07:24.974344 9540 raft_consensus.cc:2229] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:24.974905 9540 raft_consensus.cc:2258] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:24.977236 9540 ts_tablet_manager.cc:1905] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:24.979399 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9106
I20260925 15:07:24.984531 9540 ts_tablet_manager.cc:1918] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:24.984761 9540 log.cc:1196] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/bb9c8add76104eca8ece54b3ca57fcef
I20260925 15:07:24.985426 9540 ts_tablet_manager.cc:1939] T bb9c8add76104eca8ece54b3ca57fcef P 6d2eac08a2564ad99c2f51c0ac52c10c: Deleting consensus metadata
I20260925 15:07:24.987242 9029 catalog_manager.cc:4841] TS 6d2eac08a2564ad99c2f51c0ac52c10c (127.4.205.194:33069): tablet bb9c8add76104eca8ece54b3ca57fcef (table test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]) successfully deleted
I20260925 15:07:24.990681 9542 tablet_replica.cc:330] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: stopping tablet replica
I20260925 15:07:24.991165 9542 raft_consensus.cc:2229] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:24.991806 9542 raft_consensus.cc:2258] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:24.992547 9541 tablet_replica.cc:330] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: stopping tablet replica
I20260925 15:07:24.992947 9541 raft_consensus.cc:2229] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:24.993468 9541 raft_consensus.cc:2258] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:24.994133 9542 ts_tablet_manager.cc:1905] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:24.995371 9541 ts_tablet_manager.cc:1905] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:07:24 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:25.001703 9542 ts_tablet_manager.cc:1918] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:25.001950 9542 log.cc:1196] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/bb9c8add76104eca8ece54b3ca57fcef
I20260925 15:07:25.002573 9542 ts_tablet_manager.cc:1939] T bb9c8add76104eca8ece54b3ca57fcef P 2adffe60e98a47ec914a34c5aa5af33e: Deleting consensus metadata
I20260925 15:07:25.002889 9541 ts_tablet_manager.cc:1918] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:25.003067 9541 log.cc:1196] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/bb9c8add76104eca8ece54b3ca57fcef
I20260925 15:07:25.003563 9541 ts_tablet_manager.cc:1939] T bb9c8add76104eca8ece54b3ca57fcef P b942b1777b904e318fa4fe8b9979bd3f: Deleting consensus metadata
I20260925 15:07:25.005200 9029 catalog_manager.cc:4841] TS b942b1777b904e318fa4fe8b9979bd3f (127.4.205.195:35131): tablet bb9c8add76104eca8ece54b3ca57fcef (table test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]) successfully deleted
I20260925 15:07:25.005280 9030 catalog_manager.cc:4841] TS 2adffe60e98a47ec914a34c5aa5af33e (127.4.205.193:42675): tablet bb9c8add76104eca8ece54b3ca57fcef (table test-table [id=087b1a8103dd408ca3eb7283bc23ebfc]) successfully deleted
I20260925 15:07:25.008347 9513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:24.981140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.4.205.193:42675)
0925 15:07:24.982228 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:24.982235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:24.990072 (+ 7837us) server_negotiation.cc:407] Connection header received
0925 15:07:24.990259 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:24.990266 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:24.990394 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:24.990508 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:24.992489 (+ 1981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.993572 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.995666 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:24.996399 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:24.996523 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:25.001595 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:25.001608 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:25.001624 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:25.001679 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:25.005762 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.006580 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.006586 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.006588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.006709 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.007092 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.007095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.007097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.007389 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:25.007601 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:25.007894 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:07:25.008053 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":980}
I20260925 15:07:25.009047 9228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:25.102283 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9106
I20260925 15:07:25.119194 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9239
Sep 25 15:07:25 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:25.132649 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:25.119883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35140 (local address 127.4.205.194:33069)
0925 15:07:25.120061 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:25.120065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:25.120082 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:25.120213 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:25.120217 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:25.120280 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:25.120361 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:25.121483 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.122258 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.123396 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.123763 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.123860 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:25.127664 (+ 3804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:25.127675 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:25.127689 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:25.127733 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:25.130473 (+ 2740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.131015 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.131022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.131025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.131140 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.131439 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.131442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.131444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.131748 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:25.131963 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:25.132143 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 15:07:25.132299 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:07:25.133271 9361 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:25.219117 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9239
I20260925 15:07:25.239241 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9371
Sep 25 15:07:25 dist-test-slave-64fx krb5kdc[9001](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:25.253261 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:25.239978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.4.205.195:35131)
0925 15:07:25.240157 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:25.240161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:25.240327 (+ 166us) server_negotiation.cc:407] Connection header received
0925 15:07:25.240536 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:25.240546 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:25.240605 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:25.240703 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:25.242084 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.242807 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.244109 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:25.244483 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:25.244605 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:25.248693 (+ 4088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:25.248705 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:25.248721 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:25.248759 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:25.251218 (+ 2459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.251762 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.251769 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.251771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.251886 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:25.252160 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:25.252163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:25.252165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:25.252478 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:25.252666 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:25.252897 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:07:25.253039 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:07:25.253862 9493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:25.338338 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9371
I20260925 15:07:25.353847 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9014
I20260925 15:07:25.354908 9075 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:25.456750 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9014
2026-09-25T15:07:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (3872 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T15:07:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:25Z Disabled control of system clock
I20260925 15:07:25.517150 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41043
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:07:25.809852 9551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:25.810124 9551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:25.810298 9551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:25.817250 9551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:25.817341 9551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:25.817410 9551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:25.817507 9551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:25.817585 9551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:25.826191 9551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41043
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:25.827410 9551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:25.829078 9551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:25.840250 9557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:25.840317 9556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:25.840345 9559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:25.841333 9551 server_base.cc:1034] running on GCE node
I20260925 15:07:25.842053 9551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:25.843343 9551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:25.844527 9551 hybrid_clock.cc:648] HybridClock initialized: now 1790348845844494 us; error 32 us; skew 500 ppm
I20260925 15:07:25.844848 9551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:25.846917 9551 webserver.cc:460] Webserver started at http://127.4.205.254:44067/ using document root <none> and password file <none>
I20260925 15:07:25.847546 9551 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:25.847653 9551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:25.847975 9551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:25.850689 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "1660ba32dc9642c6ab7e90995ed37a24"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-64fx"
server_key: "e9d88a6bcaefa2d3d031807c11201f15"
server_key_iv: "1bf6e7cac1f8bd81f6003f12f69304e9"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.851454 9551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "1660ba32dc9642c6ab7e90995ed37a24"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-64fx"
server_key: "e9d88a6bcaefa2d3d031807c11201f15"
server_key_iv: "1bf6e7cac1f8bd81f6003f12f69304e9"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.856144 9551 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260925 15:07:25.859157 9565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:25.860303 9551 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:25.860463 9551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "1660ba32dc9642c6ab7e90995ed37a24"
format_stamp: "Formatted at 2026-09-25 15:07:25 on dist-test-slave-64fx"
server_key: "e9d88a6bcaefa2d3d031807c11201f15"
server_key_iv: "1bf6e7cac1f8bd81f6003f12f69304e9"
server_key_version: "encryptionkey@0"
I20260925 15:07:25.860663 9551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:25.865721 9551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:25.866596 9551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:25.866878 9551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:25.883539 9551 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41043
I20260925 15:07:25.883575 9616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41043 every 8 connection(s)
I20260925 15:07:25.884687 9551 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:25.888012 9617 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:25.890929 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9551
I20260925 15:07:25.891131 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:25.891713 4919 external_mini_cluster.cc:1428] Setting key c3f2a041e0c588f9fa1baa563b0a353f
I20260925 15:07:25.899199 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:25.893479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49356 (local address 127.4.205.254:41043)
0925 15:07:25.894304 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:25.894313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:25.894349 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:07:25.895965 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:25.895985 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:25.896382 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:25.896760 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:25.897805 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:25.897810 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:25.897821 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:25.897833 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:25.897882 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:25.897915 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:25.898119 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:25.898238 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:07:25.898449 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":301,"threads_started":1}
I20260925 15:07:25.899649 9617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: Bootstrap starting.
I20260925 15:07:25.902633 9617 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:25.903467 9617 log.cc:824] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:25.905679 9617 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: No bootstrap required, opened a new log
I20260925 15:07:25.912611 9617 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } }
I20260925 15:07:25.912874 9617 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:25.912948 9617 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1660ba32dc9642c6ab7e90995ed37a24, State: Initialized, Role: FOLLOWER
I20260925 15:07:25.913390 9617 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } }
I20260925 15:07:25.913650 9617 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:25.913807 9617 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:25.913975 9617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:25.915870 9617 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } }
I20260925 15:07:25.916311 9617 leader_election.cc:302] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1660ba32dc9642c6ab7e90995ed37a24; no voters:
I20260925 15:07:25.916834 9617 leader_election.cc:288] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:25.916970 9622 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:25.918067 9622 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [term 1 LEADER]: Becoming Leader. State: Replica: 1660ba32dc9642c6ab7e90995ed37a24, State: Running, Role: LEADER
I20260925 15:07:25.918546 9622 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } }
I20260925 15:07:25.918843 9617 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:25.921097 9624 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1660ba32dc9642c6ab7e90995ed37a24. Latest consensus state: current_term: 1 leader_uuid: "1660ba32dc9642c6ab7e90995ed37a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } } }
I20260925 15:07:25.921577 9624 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:25.921185 9623 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1660ba32dc9642c6ab7e90995ed37a24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1660ba32dc9642c6ab7e90995ed37a24" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41043 } } }
I20260925 15:07:25.921708 9623 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:25.925398 9630 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:25.930130 9630 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:25.936863 9630 catalog_manager.cc:1347] Generated new cluster ID: 8e7e153a7c8d46e6979a3cd8eda75cb1
I20260925 15:07:25.936961 9630 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:25.981081 9630 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:25.981587 9630 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:25.989530 9630 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: Generated new TSK 0
I20260925 15:07:25.990458 9630 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:26.003087 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--builtin_ntp_servers=127.4.205.212:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:07:26.313866 9641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:26.314111 9641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:26.314270 9641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:26.321224 9641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:26.321343 9641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:26.321507 9641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:26.330389 9641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:26.331629 9641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:26.333364 9641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:26.345923 9646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.345882 9647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.345945 9649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:26.347117 9641 server_base.cc:1034] running on GCE node
I20260925 15:07:26.347725 9641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:26.348626 9641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:26.349845 9641 hybrid_clock.cc:648] HybridClock initialized: now 1790348846349799 us; error 58 us; skew 500 ppm
I20260925 15:07:26.350155 9641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:26.352288 9641 webserver.cc:460] Webserver started at http://127.4.205.193:45441/ using document root <none> and password file <none>
I20260925 15:07:26.352943 9641 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:26.353070 9641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:26.353374 9641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:26.356199 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "77f0206724e141e08fa1048a1f7152a5"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "4483bb277869523435da2f123f678760"
server_key_iv: "f414239c8b1b31171e131a52388c5c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.356980 9641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "77f0206724e141e08fa1048a1f7152a5"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "4483bb277869523435da2f123f678760"
server_key_iv: "f414239c8b1b31171e131a52388c5c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.362015 9641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 15:07:26.365381 9655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.366717 9641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:26.366887 9641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "77f0206724e141e08fa1048a1f7152a5"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "4483bb277869523435da2f123f678760"
server_key_iv: "f414239c8b1b31171e131a52388c5c9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.367110 9641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:26.372429 9641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:26.373181 9641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:26.373443 9641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:26.374346 9641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:26.375646 9641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:26.375751 9641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.375871 9641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:26.375972 9641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.419334 9641 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:35595
I20260925 15:07:26.419376 9767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:35595 every 8 connection(s)
I20260925 15:07:26.420711 9641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:26.428602 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.423254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:40849 (local address 127.4.205.254:41043)
0925 15:07:26.423688 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:26.423693 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:26.424861 (+ 1168us) server_negotiation.cc:407] Connection header received
0925 15:07:26.426741 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:26.426746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:26.426804 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:26.426897 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:26.428062 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:26.428065 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:26.428069 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:26.428082 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:26.428119 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:26.428130 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:26.428217 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:26.428365 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:07:26.428407 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":250,"threads_started":1}
I20260925 15:07:26.429445 9771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.423857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41043 (local address 127.4.205.193:40849)
0925 15:07:26.424713 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:26.424748 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:26.426489 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:26.427094 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:26.427101 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:26.427188 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:26.427783 (+ 595us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:26.427785 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:26.427820 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:26.427898 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:26.428207 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:26.428213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:26.428218 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:07:26.428416 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 15:07:26.428591 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":267,"threads_started":1}
I20260925 15:07:26.430502 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9641
I20260925 15:07:26.430673 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:26.430779 9768 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41043
I20260925 15:07:26.431057 9768 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:26.431361 4919 external_mini_cluster.cc:1428] Setting key 6ea9910d5243781e1ff00538154dad4a
I20260925 15:07:26.431619 9768 heartbeater.cc:511] Master 127.4.205.254:41043 requested a full tablet report, sending...
I20260925 15:07:26.433549 9581 ts_manager.cc:194] Registered new tserver with Master: 77f0206724e141e08fa1048a1f7152a5 (127.4.205.193:35595)
I20260925 15:07:26.433970 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--builtin_ntp_servers=127.4.205.212:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 15:07:26.734679 9772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:26.734987 9772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:26.735142 9772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:26.742038 9772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:26.742147 9772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:26.742313 9772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:26.750597 9772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:26.751740 9772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:26.753429 9772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:26.766218 9777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.766184 9780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:26.766268 9778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:26.766881 9772 server_base.cc:1034] running on GCE node
I20260925 15:07:26.767373 9772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:26.768175 9772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:26.769361 9772 hybrid_clock.cc:648] HybridClock initialized: now 1790348846769328 us; error 27 us; skew 500 ppm
I20260925 15:07:26.769721 9772 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:26.771684 9772 webserver.cc:460] Webserver started at http://127.4.205.194:33641/ using document root <none> and password file <none>
I20260925 15:07:26.772279 9772 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:26.772382 9772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:26.772697 9772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:26.775334 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "493bb275e6a3477493200c2d382a4656"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "402cd13ddd56716848f697f5d3e9581d"
server_key_iv: "7224388fff504d47a420c8e67c1f3129"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.776082 9772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "493bb275e6a3477493200c2d382a4656"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "402cd13ddd56716848f697f5d3e9581d"
server_key_iv: "7224388fff504d47a420c8e67c1f3129"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.780786 9772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:07:26.783877 9786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.785069 9772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:26.785233 9772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "493bb275e6a3477493200c2d382a4656"
format_stamp: "Formatted at 2026-09-25 15:07:26 on dist-test-slave-64fx"
server_key: "402cd13ddd56716848f697f5d3e9581d"
server_key_iv: "7224388fff504d47a420c8e67c1f3129"
server_key_version: "encryptionkey@0"
I20260925 15:07:26.785435 9772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:26.790685 9772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:26.791364 9772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:26.791622 9772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:26.792482 9772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:26.793828 9772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:26.793942 9772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.794063 9772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:26.794153 9772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:26.825062 9772 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:45045
I20260925 15:07:26.825106 9898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:45045 every 8 connection(s)
I20260925 15:07:26.826303 9772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:26.830782 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9772
I20260925 15:07:26.830957 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:26.831364 4919 external_mini_cluster.cc:1428] Setting key 6a06fb17f77c5b4262dcbddff9c37237
I20260925 15:07:26.834247 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.828653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:48775 (local address 127.4.205.254:41043)
0925 15:07:26.828835 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:26.828858 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:26.830191 (+ 1333us) server_negotiation.cc:407] Connection header received
0925 15:07:26.831863 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:26.831869 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:26.831936 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:26.832041 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:26.833502 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:26.833508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:26.833512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:26.833529 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:26.833587 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:26.833602 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:26.833715 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:26.833899 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 15:07:26.833959 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:07:26.834226 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--builtin_ntp_servers=127.4.205.212:45285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 15:07:26.835047 9901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:26.829182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41043 (local address 127.4.205.194:48775)
0925 15:07:26.830058 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:26.830089 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:26.831621 (+ 1532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:26.832435 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:26.832443 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:26.832524 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:26.833196 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:26.833200 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:26.833247 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:26.833337 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:26.833694 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:26.833701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:26.833706 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:07:26.833935 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 15:07:26.834122 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":258,"threads_started":1}
I20260925 15:07:26.836304 9899 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41043
I20260925 15:07:26.836585 9899 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:26.837141 9899 heartbeater.cc:511] Master 127.4.205.254:41043 requested a full tablet report, sending...
I20260925 15:07:26.838430 9581 ts_manager.cc:194] Registered new tserver with Master: 493bb275e6a3477493200c2d382a4656 (127.4.205.194:45045)
W20260925 15:07:27.135392 9902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:27.135670 9902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:27.135831 9902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:27.142740 9902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:27.142861 9902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:27.143033 9902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:27.151526 9902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:45285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41043
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:27.152661 9902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:27.154354 9902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:27.166965 9907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:27.166929 9908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:27.167099 9910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:27.168030 9902 server_base.cc:1034] running on GCE node
I20260925 15:07:27.168514 9902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:27.169354 9902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:27.170578 9902 hybrid_clock.cc:648] HybridClock initialized: now 1790348847170523 us; error 68 us; skew 500 ppm
I20260925 15:07:27.170914 9902 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:27.173051 9902 webserver.cc:460] Webserver started at http://127.4.205.195:41509/ using document root <none> and password file <none>
I20260925 15:07:27.173694 9902 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:27.173775 9902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:27.174023 9902 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:27.176523 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "98f25ad744c94938aea00ff5b1202a4c"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-64fx"
server_key: "0ef1302afbf684b092a9e49425ec99df"
server_key_iv: "cdc53fceb9b85eb14a3f26dd3b72f55b"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.177237 9902 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "98f25ad744c94938aea00ff5b1202a4c"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-64fx"
server_key: "0ef1302afbf684b092a9e49425ec99df"
server_key_iv: "cdc53fceb9b85eb14a3f26dd3b72f55b"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.182073 9902 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:07:27.185042 9916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.186259 9902 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:27.186406 9902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "98f25ad744c94938aea00ff5b1202a4c"
format_stamp: "Formatted at 2026-09-25 15:07:27 on dist-test-slave-64fx"
server_key: "0ef1302afbf684b092a9e49425ec99df"
server_key_iv: "cdc53fceb9b85eb14a3f26dd3b72f55b"
server_key_version: "encryptionkey@0"
I20260925 15:07:27.186617 9902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:27.192015 9902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:27.193018 9902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:27.193275 9902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:27.194401 9902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:27.196072 9902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:27.196182 9902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.196280 9902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:27.196377 9902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:27.227700 9902 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:39123
I20260925 15:07:27.227756 10028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:39123 every 8 connection(s)
I20260925 15:07:27.229018 9902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:27.230578 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 9902
I20260925 15:07:27.230742 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:27.231171 4919 external_mini_cluster.cc:1428] Setting key 24db1a00d1dcae9ab883cebe0fc6b3f5
I20260925 15:07:27.237373 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.231563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:51741 (local address 127.4.205.254:41043)
0925 15:07:27.231744 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.231748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.233580 (+ 1832us) server_negotiation.cc:407] Connection header received
0925 15:07:27.235259 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.235264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.235334 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.235447 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.236644 (+ 1197us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.236650 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.236661 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.236678 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.236735 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.236747 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.236862 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.237110 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.237160 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:07:27.238291 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.232188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41043 (local address 127.4.205.195:51741)
0925 15:07:27.233415 (+ 1227us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.233455 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.235013 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.235662 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.235669 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.235753 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.236373 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.236375 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.236408 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.236481 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.236849 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.236859 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.236868 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.237172 (+ 304us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.237402 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":764,"thread_start_us":576,"threads_started":1}
I20260925 15:07:27.239482 10029 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41043
I20260925 15:07:27.239715 10029 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:27.240177 10029 heartbeater.cc:511] Master 127.4.205.254:41043 requested a full tablet report, sending...
I20260925 15:07:27.241410 9581 ts_manager.cc:194] Registered new tserver with Master: 98f25ad744c94938aea00ff5b1202a4c (127.4.205.195:39123)
I20260925 15:07:27.246088 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:27.252507 9770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.250361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49364 (local address 127.4.205.254:41043)
0925 15:07:27.250536 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.250545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.250980 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:07:27.251096 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.251099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.251143 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.251214 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.251972 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.251976 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.251979 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.251991 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.252020 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.252030 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.252111 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.252184 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.252224 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:07:27.255766 9581 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:27.262456 9581 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:27.280668 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.273484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:45045 (local address 127.0.0.1:42172)
0925 15:07:27.274912 (+ 1428us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.274945 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.275066 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.276389 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.276401 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.276513 (+ 112us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.277495 (+ 982us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.277501 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.277537 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.277649 (+ 112us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.279739 (+ 2090us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.279753 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.279764 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.279978 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.280120 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1167,"thread_start_us":4346,"threads_started":1}
I20260925 15:07:27.282228 10041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.273996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55602)
0925 15:07:27.277993 (+ 3997us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.278010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.278098 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.279776 (+ 1678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.279784 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.279813 (+ 29us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.280715 (+ 902us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.280720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.280740 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.280793 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.281375 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.281382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.281390 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.281588 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.281650 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3860,"thread_start_us":664,"threads_started":1}
I20260925 15:07:27.283022 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.274474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55602 (local address 127.4.205.193:35595)
0925 15:07:27.278137 (+ 3663us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.278154 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.278179 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:27.278243 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.278252 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.278524 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.278695 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.280986 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.280998 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.281015 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.281047 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.281111 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.281134 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.281315 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.282508 (+ 1193us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.282620 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3504,"thread_start_us":3392,"threads_started":1}
I20260925 15:07:27.283056 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.273733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42172 (local address 127.4.205.194:45045)
0925 15:07:27.275670 (+ 1937us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.275691 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.275749 (+ 58us) server_negotiation.cc:407] Connection header received
0925 15:07:27.275831 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.275842 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.276130 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.276308 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.279435 (+ 3127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.279447 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.279478 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.279498 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.279561 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.279584 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.282464 (+ 2880us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.282528 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.282618 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1736,"thread_start_us":1542,"threads_started":1}
I20260925 15:07:27.286501 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.278200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:39123 (local address 127.0.0.1:59062)
0925 15:07:27.280870 (+ 2670us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.280885 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.280964 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.284656 (+ 3692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.284661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.284691 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.285165 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.285170 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.285185 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.285222 (+ 37us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.286125 (+ 903us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.286129 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.286134 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.286207 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.286251 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2588,"thread_start_us":5455,"threads_started":1}
I20260925 15:07:27.287258 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.278555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59062 (local address 127.4.205.195:39123)
0925 15:07:27.284205 (+ 5650us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.284212 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.284240 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:07:27.284305 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.284312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.284491 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.284634 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.285905 (+ 1271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.285912 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.285924 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.285940 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.285982 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.285999 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.286757 (+ 758us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.286819 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.286895 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5511,"thread_start_us":5362,"threads_started":1}
I20260925 15:07:27.286927 9703 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 (DEFAULT_TABLE table=test-table [id=22742be7731e4d5dbae9e65f6a314824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.287179 9834 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 (DEFAULT_TABLE table=test-table [id=22742be7731e4d5dbae9e65f6a314824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.288187 9703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.288403 9834 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.298553 9964 tablet_service.cc:1467] Processing CreateTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 (DEFAULT_TABLE table=test-table [id=22742be7731e4d5dbae9e65f6a314824]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:27.299841 10048 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Bootstrap starting.
I20260925 15:07:27.299966 9964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:27.300177 10047 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Bootstrap starting.
I20260925 15:07:27.303561 10048 tablet_bootstrap.cc:654] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.304695 10047 tablet_bootstrap.cc:654] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.304755 10048 log.cc:824] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.305987 10047 log.cc:824] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.308310 10048 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: No bootstrap required, opened a new log
I20260925 15:07:27.308580 10048 ts_tablet_manager.cc:1397] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20260925 15:07:27.308774 10047 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: No bootstrap required, opened a new log
I20260925 15:07:27.309023 10047 ts_tablet_manager.cc:1397] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260925 15:07:27.311494 10051 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Bootstrap starting.
I20260925 15:07:27.315161 10051 tablet_bootstrap.cc:654] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:27.316264 10051 log.cc:824] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:27.317195 10047 raft_consensus.cc:348] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.317653 10047 raft_consensus.cc:374] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.317741 10047 raft_consensus.cc:729] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77f0206724e141e08fa1048a1f7152a5, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.317890 10048 raft_consensus.cc:348] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.318270 10048 raft_consensus.cc:374] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.318372 10048 raft_consensus.cc:729] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 493bb275e6a3477493200c2d382a4656, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.318349 10047 consensus_queue.cc:260] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.319026 10048 consensus_queue.cc:260] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.319469 10051 tablet_bootstrap.cc:492] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: No bootstrap required, opened a new log
I20260925 15:07:27.319515 9768 heartbeater.cc:503] Master 127.4.205.254:41043 was elected leader, sending a full tablet report...
I20260925 15:07:27.319757 10051 ts_tablet_manager.cc:1397] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260925 15:07:27.320022 10047 ts_tablet_manager.cc:1428] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260925 15:07:27.321025 9899 heartbeater.cc:503] Master 127.4.205.254:41043 was elected leader, sending a full tablet report...
I20260925 15:07:27.322914 10048 ts_tablet_manager.cc:1428] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Time spent starting tablet: real 0.014s user 0.012s sys 0.001s
W20260925 15:07:27.328024 9900 tablet.cc:2369] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:27.328619 10051 raft_consensus.cc:348] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.328982 10051 raft_consensus.cc:374] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:27.329056 10051 raft_consensus.cc:729] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f25ad744c94938aea00ff5b1202a4c, State: Initialized, Role: FOLLOWER
I20260925 15:07:27.329466 10051 consensus_queue.cc:260] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.330366 10029 heartbeater.cc:503] Master 127.4.205.254:41043 was elected leader, sending a full tablet report...
I20260925 15:07:27.330894 10051 ts_tablet_manager.cc:1428] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260925 15:07:27.364792 10055 raft_consensus.cc:482] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:27.365010 10055 raft_consensus.cc:504] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.366595 10055 leader_election.cc:288] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 493bb275e6a3477493200c2d382a4656 (127.4.205.194:45045), 77f0206724e141e08fa1048a1f7152a5 (127.4.205.193:35595)
I20260925 15:07:27.368981 10031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.366832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.4.205.195:48227)
0925 15:07:27.366983 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.367002 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.367094 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.367529 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.367534 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.367561 (+ 27us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.368218 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.368222 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.368240 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.368289 (+ 49us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.368607 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.368612 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.368618 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.368715 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.368765 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:07:27.369074 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.367014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:48227 (local address 127.4.205.193:35595)
0925 15:07:27.367202 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.367211 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.367237 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:07:27.367306 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.367313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.367372 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.367484 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.368420 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.368426 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.368432 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.368452 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.368488 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.368504 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.368609 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.368718 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.368788 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:07:27.369413 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.366875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:58799 (local address 127.4.205.194:45045)
0925 15:07:27.367025 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.367033 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.367628 (+ 595us) server_negotiation.cc:407] Connection header received
0925 15:07:27.367754 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.367758 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.367829 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.367943 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.368811 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.368815 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.368819 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.368833 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.368865 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.368878 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.368985 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.369125 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.369177 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:07:27.369985 10056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.366832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:45045 (local address 127.4.205.195:58799)
0925 15:07:27.367531 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.367546 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:27.367646 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:27.367967 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:27.367970 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:27.367986 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:27.368640 (+ 654us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:07:27.368644 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:27.368659 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:07:27.368697 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:27.368978 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:27.368983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:27.368988 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:07:27.369624 (+ 636us) client_negotiation.cc:240] Negotiation successful
0925 15:07:27.369672 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"mutex_wait_us":134,"thread_start_us":274,"threads_started":1}
I20260925 15:07:27.369982 9723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3414a4f6f41e5bf69ce7b5e8c1da0" candidate_uuid: "98f25ad744c94938aea00ff5b1202a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f0206724e141e08fa1048a1f7152a5" is_pre_election: true
I20260925 15:07:27.370505 9723 raft_consensus.cc:2454] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f25ad744c94938aea00ff5b1202a4c in term 0.
I20260925 15:07:27.370808 9854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3414a4f6f41e5bf69ce7b5e8c1da0" candidate_uuid: "98f25ad744c94938aea00ff5b1202a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493bb275e6a3477493200c2d382a4656" is_pre_election: true
I20260925 15:07:27.371173 9919 leader_election.cc:302] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77f0206724e141e08fa1048a1f7152a5, 98f25ad744c94938aea00ff5b1202a4c; no voters:
I20260925 15:07:27.371248 9854 raft_consensus.cc:2454] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98f25ad744c94938aea00ff5b1202a4c in term 0.
I20260925 15:07:27.371527 10055 raft_consensus.cc:2781] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:27.371675 10055 raft_consensus.cc:482] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:27.371784 10055 raft_consensus.cc:3037] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.374089 10055 raft_consensus.cc:504] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.374648 10055 leader_election.cc:288] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [CANDIDATE]: Term 1 election: Requested vote from peers 493bb275e6a3477493200c2d382a4656 (127.4.205.194:45045), 77f0206724e141e08fa1048a1f7152a5 (127.4.205.193:35595)
I20260925 15:07:27.374929 9854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3414a4f6f41e5bf69ce7b5e8c1da0" candidate_uuid: "98f25ad744c94938aea00ff5b1202a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "493bb275e6a3477493200c2d382a4656"
I20260925 15:07:27.375123 9854 raft_consensus.cc:3037] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.375068 9723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fc3414a4f6f41e5bf69ce7b5e8c1da0" candidate_uuid: "98f25ad744c94938aea00ff5b1202a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77f0206724e141e08fa1048a1f7152a5"
I20260925 15:07:27.375288 9723 raft_consensus.cc:3037] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:27.377137 9854 raft_consensus.cc:2454] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f25ad744c94938aea00ff5b1202a4c in term 1.
I20260925 15:07:27.377359 9723 raft_consensus.cc:2454] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 98f25ad744c94938aea00ff5b1202a4c in term 1.
I20260925 15:07:27.377480 9917 leader_election.cc:302] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 493bb275e6a3477493200c2d382a4656, 98f25ad744c94938aea00ff5b1202a4c; no voters:
I20260925 15:07:27.377844 10055 raft_consensus.cc:2781] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:27.378234 10055 raft_consensus.cc:686] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 1 LEADER]: Becoming Leader. State: Replica: 98f25ad744c94938aea00ff5b1202a4c, State: Running, Role: LEADER
I20260925 15:07:27.378654 10055 consensus_queue.cc:237] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } }
I20260925 15:07:27.382879 9581 catalog_manager.cc:5495] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to 98f25ad744c94938aea00ff5b1202a4c (127.4.205.195). New cstate: current_term: 1 leader_uuid: "98f25ad744c94938aea00ff5b1202a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "98f25ad744c94938aea00ff5b1202a4c" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:27.402362 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.400168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59078 (local address 127.4.205.195:39123)
0925 15:07:27.400338 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.400344 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.400465 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:07:27.400679 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.400683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.400746 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.400845 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.401794 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.401801 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.401807 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.401824 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.401866 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.401877 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.401999 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.402077 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.402129 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:07:27.409247 9854 raft_consensus.cc:1261] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 1 FOLLOWER]: Refusing update from remote peer 98f25ad744c94938aea00ff5b1202a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:27.409268 9723 raft_consensus.cc:1261] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 1 FOLLOWER]: Refusing update from remote peer 98f25ad744c94938aea00ff5b1202a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:27.410095 10057 consensus_queue.cc:1035] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "77f0206724e141e08fa1048a1f7152a5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 35595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:27.410394 10055 consensus_queue.cc:1035] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "493bb275e6a3477493200c2d382a4656" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:27.421222 10060 mvcc.cc:205] Tried to move back new op lower bound from 7333268878979977216 to 7333268878866104320. Current Snapshot: MvccSnapshot[applied={T|T < 7333268878979977216}]
I20260925 15:07:27.421487 10062 mvcc.cc:205] Tried to move back new op lower bound from 7333268878979977216 to 7333268878866104320. Current Snapshot: MvccSnapshot[applied={T|T < 7333268878979977216}]
I20260925 15:07:27.450416 9581 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:27.450659 9581 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:27.453744 9581 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1660ba32dc9642c6ab7e90995ed37a24: Sending DeleteTablet for 3 replicas of tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0
I20260925 15:07:27.454685 9834 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='slave'} at 127.0.0.1:42172
I20260925 15:07:27.454766 9964 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='slave'} at 127.0.0.1:59062
I20260925 15:07:27.455000 9703 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:27 UTC) from {username='slave'} at 127.0.0.1:55602
I20260925 15:07:27.457369 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9641
I20260925 15:07:27.486478 10072 tablet_replica.cc:330] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: stopping tablet replica
I20260925 15:07:27.487015 10072 raft_consensus.cc:2229] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:27.487571 10072 raft_consensus.cc:2258] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.489266 10044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.458713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55604 (local address 127.4.205.193:35595)
0925 15:07:27.458946 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.458955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.484696 (+ 25741us) server_negotiation.cc:407] Connection header received
0925 15:07:27.485166 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.485174 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.485288 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.485372 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.488346 (+ 2974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.488354 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.488361 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.488385 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.488431 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.488451 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.488569 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.488838 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.488900 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 15:07:27.490397 9763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.490686 10070 tablet_replica.cc:330] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: stopping tablet replica
I20260925 15:07:27.491096 10070 raft_consensus.cc:2229] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:27.491566 10070 raft_consensus.cc:2258] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.492377 10071 tablet_replica.cc:330] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: stopping tablet replica
I20260925 15:07:27.492774 10071 raft_consensus.cc:2229] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:27.493516 10071 raft_consensus.cc:2258] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:27.495335 10071 ts_tablet_manager.cc:1905] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.495347 10070 ts_tablet_manager.cc:1905] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.501104 10070 ts_tablet_manager.cc:1918] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:27.501259 10070 log.cc:1196] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/9fc3414a4f6f41e5bf69ce7b5e8c1da0
I20260925 15:07:27.501811 10070 ts_tablet_manager.cc:1939] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 493bb275e6a3477493200c2d382a4656: Deleting consensus metadata
I20260925 15:07:27.502526 10071 ts_tablet_manager.cc:1918] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:27.502735 10071 log.cc:1196] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/9fc3414a4f6f41e5bf69ce7b5e8c1da0
I20260925 15:07:27.503329 10071 ts_tablet_manager.cc:1939] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 98f25ad744c94938aea00ff5b1202a4c: Deleting consensus metadata
I20260925 15:07:27.503513 9566 catalog_manager.cc:4841] TS 493bb275e6a3477493200c2d382a4656 (127.4.205.194:45045): tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 (table test-table [id=22742be7731e4d5dbae9e65f6a314824]) successfully deleted
I20260925 15:07:27.504962 9566 catalog_manager.cc:4841] TS 98f25ad744c94938aea00ff5b1202a4c (127.4.205.195:39123): tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0 (table test-table [id=22742be7731e4d5dbae9e65f6a314824]) successfully deleted
I20260925 15:07:27.490993 10072 ts_tablet_manager.cc:1905] T 9fc3414a4f6f41e5bf69ce7b5e8c1da0 P 77f0206724e141e08fa1048a1f7152a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:27.580178 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9641
W20260925 15:07:27.596271 9568 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.205.193:35595 (error 108)
I20260925 15:07:27.597649 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9772
I20260925 15:07:27.598313 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.597711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55608)
0925 15:07:27.597977 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.598121 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
W20260925 15:07:27.598542 9568 catalog_manager.cc:4584] TS 77f0206724e141e08fa1048a1f7152a5 (127.4.205.193:35595): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9fc3414a4f6f41e5bf69ce7b5e8c1da0: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
I20260925 15:07:27.600713 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.598386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42182 (local address 127.4.205.194:45045)
0925 15:07:27.598547 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.598554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.598905 (+ 351us) server_negotiation.cc:407] Connection header received
0925 15:07:27.599083 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.599087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.599171 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.599250 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.600097 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.600101 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.600104 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.600119 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.600148 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.600159 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.600252 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.600376 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.600418 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:27.601341 9894 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.652082 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.651564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55622)
0925 15:07:27.651821 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.651919 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:07:27.653153 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.652762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55636)
0925 15:07:27.652914 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.653007 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:07:27.686909 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9772
I20260925 15:07:27.704736 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9902
I20260925 15:07:27.707834 10046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.705682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.4.205.195:39123)
0925 15:07:27.705878 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:27.705883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:27.705902 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:07:27.706057 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:27.706061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:27.706112 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:27.706221 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:27.707131 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:27.707138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:27.707150 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:27.707164 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:27.707192 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:27.707203 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:27.707344 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:27.707490 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:07:27.707544 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:07:27.708603 10024 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.713796 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.713304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55638)
0925 15:07:27.713484 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.713615 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:07:27.714738 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.714405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55646)
0925 15:07:27.714552 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.714625 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260925 15:07:27.795325 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.794851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55652)
0925 15:07:27.795074 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.795161 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 15:07:27.796337 10043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:27.795977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:35595 (local address 127.0.0.1:55662)
0925 15:07:27.796161 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:27.796226 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:35595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260925 15:07:27.824555 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9902
I20260925 15:07:27.843111 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 9551
I20260925 15:07:27.844173 9612 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:27.957509 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 9551
2026-09-25T15:07:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2497 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 15:07:27.994189 4919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:07:27.996649 4919 webserver.cc:460] Webserver started at https://127.0.0.1:45605/ using document root <none> and password file <none>
I20260925 15:07:27.996973 4919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:07:27.997541 4919 webserver.cc:460] Webserver started at http://127.0.0.1:36137/ using document root <none> and password file <none>
2026-09-25T15:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:28Z Disabled control of system clock
I20260925 15:07:28.021237 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:34403
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:40519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:34403
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/testchainca.cert with env {}
W20260925 15:07:28.321703 10084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:28.321979 10084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:28.322067 10084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:28.328877 10084 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:28.328969 10084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:28.329005 10084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:28.329051 10084 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:28.329085 10084 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:07:28.329144 10084 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:07:28.329182 10084 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45605/jwks.json
I20260925 15:07:28.338637 10084 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:34403
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:34403
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:28.339718 10084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:28.340880 10084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:28.352270 10089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.352234 10090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:28.352324 10092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:28.353226 10084 server_base.cc:1034] running on GCE node
I20260925 15:07:28.354009 10084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:28.355161 10084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:28.356349 10084 hybrid_clock.cc:648] HybridClock initialized: now 1790348848356320 us; error 26 us; skew 500 ppm
I20260925 15:07:28.356667 10084 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:28.358762 10084 webserver.cc:460] Webserver started at http://127.4.205.254:44759/ using document root <none> and password file <none>
I20260925 15:07:28.359388 10084 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:28.359503 10084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:28.359787 10084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:28.362567 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "a7d8d20dd76e47258236beb9b2554a02"
format_stamp: "Formatted at 2026-09-25 15:07:28 on dist-test-slave-64fx"
server_key: "b367cc1f2956159c9571566bbc4ef740"
server_key_iv: "dda84280d8a1694a2bcdaf9bc84b6581"
server_key_version: "encryptionkey@0"
I20260925 15:07:28.363372 10084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "a7d8d20dd76e47258236beb9b2554a02"
format_stamp: "Formatted at 2026-09-25 15:07:28 on dist-test-slave-64fx"
server_key: "b367cc1f2956159c9571566bbc4ef740"
server_key_iv: "dda84280d8a1694a2bcdaf9bc84b6581"
server_key_version: "encryptionkey@0"
I20260925 15:07:28.368006 10084 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:28.371191 10098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:28.372447 10084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:28.372618 10084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "a7d8d20dd76e47258236beb9b2554a02"
format_stamp: "Formatted at 2026-09-25 15:07:28 on dist-test-slave-64fx"
server_key: "b367cc1f2956159c9571566bbc4ef740"
server_key_iv: "dda84280d8a1694a2bcdaf9bc84b6581"
server_key_version: "encryptionkey@0"
I20260925 15:07:28.372812 10084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:28.386584 10084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:28.387293 10084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:28.387579 10084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:28.408286 10084 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:34403
I20260925 15:07:28.408294 10149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:34403 every 8 connection(s)
I20260925 15:07:28.409546 10084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:28.412956 10150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:28.417140 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10084
I20260925 15:07:28.417359 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:28.418155 4919 external_mini_cluster.cc:1428] Setting key 994de635037c3fb6bf5b7c419664dd6a
I20260925 15:07:28.423882 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02: Bootstrap starting.
I20260925 15:07:28.427191 10150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:28.428344 10150 log.cc:824] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:28.431121 10150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02: No bootstrap required, opened a new log
I20260925 15:07:28.431285 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:28.419882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.4.205.254:34403)
0925 15:07:28.420713 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:28.420722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:28.420762 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:07:28.422631 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:28.422651 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:28.423074 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:28.423356 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:28.423936 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.425164 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.426986 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.427675 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.427796 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:28.429070 (+ 1274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:28.429082 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:28.429102 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:28.429771 (+ 669us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:28.429841 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:28.429861 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:28.429947 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:28.430150 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:07:28.430446 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":475,"thread_start_us":299,"threads_started":1}
I20260925 15:07:28.434861 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:28.439497 10150 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } }
I20260925 15:07:28.439881 10150 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:28.439998 10150 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7d8d20dd76e47258236beb9b2554a02, State: Initialized, Role: FOLLOWER
I20260925 15:07:28.440537 10150 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } }
I20260925 15:07:28.440752 10150 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:28.440887 10150 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:28.441064 10150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:28.443153 10150 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } }
I20260925 15:07:28.443683 10150 leader_election.cc:302] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7d8d20dd76e47258236beb9b2554a02; no voters:
I20260925 15:07:28.444203 10150 leader_election.cc:288] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:28.444327 10155 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:28.445034 10155 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [term 1 LEADER]: Becoming Leader. State: Replica: a7d8d20dd76e47258236beb9b2554a02, State: Running, Role: LEADER
I20260925 15:07:28.445598 10155 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } }
I20260925 15:07:28.446265 10150 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:28.448271 10157 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7d8d20dd76e47258236beb9b2554a02. Latest consensus state: current_term: 1 leader_uuid: "a7d8d20dd76e47258236beb9b2554a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } } }
I20260925 15:07:28.448613 10157 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:28.448362 10156 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7d8d20dd76e47258236beb9b2554a02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7d8d20dd76e47258236beb9b2554a02" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 34403 } } }
I20260925 15:07:28.448931 10156 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:28.450625 10160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:28.454950 10160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:28.462203 10160 catalog_manager.cc:1347] Generated new cluster ID: 2b2cef35c9f9469293e69f5858c48dff
I20260925 15:07:28.462318 10160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:28.479489 10160 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:28.481113 10160 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:28.493705 10160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7d8d20dd76e47258236beb9b2554a02: Generated new TSK 0
I20260925 15:07:28.494585 10160 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:07:28.530694 10180 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 15:07:28.530839 10180 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:07:28.531867 10153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:28.511208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33110 (local address 127.4.205.254:34403)
0925 15:07:28.511394 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:28.511408 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:28.511912 (+ 504us) server_negotiation.cc:407] Connection header received
0925 15:07:28.512158 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:28.512168 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:28.512252 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:28.512424 (+ 172us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:07:28.512844 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.513980 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.515262 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.515926 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.516053 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:28.516482 (+ 429us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:07:28.531274 (+ 14792us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:07:28.531591 (+ 317us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:33110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":61}
W20260925 15:07:28.532002 10153 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:33110: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 15:07:28.533253 10179 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:28.511133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:34403 (local address 127.0.0.1:33110)
0925 15:07:28.511819 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:28.511834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:28.511964 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:28.512425 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:28.512428 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:28.512437 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:07:28.512716 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:28.512723 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.514093 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.514096 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:28.515131 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:28.515141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:28.516080 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:28.516084 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:28.516193 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:28.516244 (+ 51us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:07:28.531492 (+ 15248us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 15:07:28.531712 (+ 220us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.205.254:34403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":375,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.4.205.254:34403: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 15:07:28.535143 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10084
I20260925 15:07:28.536077 10145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:28.623420 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10084
2026-09-25T15:07:28Z chronyd exiting
I20260925 15:07:28.672721 4919 test_util.cc:175] -----------------------------------------------
I20260925 15:07:28.672853 4919 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348808911781-4919-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (682 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 15:07:28.676355 4919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:07:28.678198 4919 webserver.cc:460] Webserver started at https://127.0.0.1:46705/ using document root <none> and password file <none>
I20260925 15:07:28.678431 4919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:07:28.679018 4919 webserver.cc:460] Webserver started at http://127.0.0.1:36471/ using document root <none> and password file <none>
2026-09-25T15:07:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:28Z Disabled control of system clock
I20260925 15:07:28.703668 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33261
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:44735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33261
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/testchainca.cert with env {}
W20260925 15:07:29.004063 10187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:29.004374 10187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:29.004556 10187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:29.011368 10187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:29.011502 10187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:29.011595 10187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:29.011698 10187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:29.011785 10187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:07:29.011870 10187 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:07:29.011948 10187 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46705/jwks.json
I20260925 15:07:29.021116 10187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33261
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:29.022315 10187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:29.023569 10187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:29.034911 10192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.034874 10195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.034934 10193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:29.035712 10187 server_base.cc:1034] running on GCE node
I20260925 15:07:29.036413 10187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:29.037727 10187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:29.038919 10187 hybrid_clock.cc:648] HybridClock initialized: now 1790348849038868 us; error 48 us; skew 500 ppm
I20260925 15:07:29.039227 10187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:29.041224 10187 webserver.cc:460] Webserver started at http://127.4.205.254:34779/ using document root <none> and password file <none>
I20260925 15:07:29.041899 10187 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:29.042011 10187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:29.042318 10187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:29.044915 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "af333a52ed894fba9da1c32174090947"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "ed55885d3a265491d29136759e6545a7"
server_key_iv: "25d90342a4cba52ac61c669dc1f06e74"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.045691 10187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "af333a52ed894fba9da1c32174090947"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "ed55885d3a265491d29136759e6545a7"
server_key_iv: "25d90342a4cba52ac61c669dc1f06e74"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.050205 10187 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:07:29.053159 10201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.054391 10187 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:07:29.054562 10187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "af333a52ed894fba9da1c32174090947"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "ed55885d3a265491d29136759e6545a7"
server_key_iv: "25d90342a4cba52ac61c669dc1f06e74"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.054749 10187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:29.072116 10187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:29.072786 10187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:29.073069 10187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:29.090159 10187 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33261
I20260925 15:07:29.090193 10252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33261 every 8 connection(s)
I20260925 15:07:29.091416 10187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:29.094782 10253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:29.099269 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10187
I20260925 15:07:29.099462 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:29.100060 4919 external_mini_cluster.cc:1428] Setting key c77fa277100c7ebbf8bb1c5fb44f6f8d
I20260925 15:07:29.105003 10253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947: Bootstrap starting.
I20260925 15:07:29.108482 10253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:29.109665 10253 log.cc:824] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:29.112668 10253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947: No bootstrap required, opened a new log
I20260925 15:07:29.113533 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.101839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54394 (local address 127.4.205.254:33261)
0925 15:07:29.103016 (+ 1177us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.103027 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.103070 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:07:29.104577 (+ 1507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.104596 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.105015 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.105371 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.105909 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.107208 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.109094 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.109883 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.110006 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.111368 (+ 1362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.111375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.111387 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:29.111889 (+ 502us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.111960 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:29.111980 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.112063 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.112321 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.112634 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":481,"threads_started":1}
I20260925 15:07:29.117231 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:29.120823 10253 raft_consensus.cc:348] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } }
I20260925 15:07:29.121176 10253 raft_consensus.cc:374] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:29.121289 10253 raft_consensus.cc:729] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af333a52ed894fba9da1c32174090947, State: Initialized, Role: FOLLOWER
I20260925 15:07:29.121853 10253 consensus_queue.cc:260] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } }
I20260925 15:07:29.122089 10253 raft_consensus.cc:388] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:29.122222 10253 raft_consensus.cc:482] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:29.122397 10253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:29.124421 10253 raft_consensus.cc:504] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } }
I20260925 15:07:29.124871 10253 leader_election.cc:302] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af333a52ed894fba9da1c32174090947; no voters:
I20260925 15:07:29.125397 10253 leader_election.cc:288] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:29.125586 10258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:29.126747 10258 raft_consensus.cc:686] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [term 1 LEADER]: Becoming Leader. State: Replica: af333a52ed894fba9da1c32174090947, State: Running, Role: LEADER
I20260925 15:07:29.127198 10258 consensus_queue.cc:237] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } }
I20260925 15:07:29.127378 10253 sys_catalog.cc:564] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:29.129619 10259 sys_catalog.cc:455] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af333a52ed894fba9da1c32174090947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } } }
I20260925 15:07:29.130028 10259 sys_catalog.cc:458] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:29.129705 10260 sys_catalog.cc:455] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af333a52ed894fba9da1c32174090947. Latest consensus state: current_term: 1 leader_uuid: "af333a52ed894fba9da1c32174090947" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af333a52ed894fba9da1c32174090947" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33261 } } }
I20260925 15:07:29.130508 10260 sys_catalog.cc:458] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:29.131047 10265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:29.136958 10265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:29.145094 10265 catalog_manager.cc:1347] Generated new cluster ID: aa7d0a1f5ba14b5592f9f4e9f0360707
I20260925 15:07:29.145193 10265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:29.168231 10265 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:29.169626 10265 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:29.180357 10265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af333a52ed894fba9da1c32174090947: Generated new TSK 0
I20260925 15:07:29.181113 10265 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 15:07:29.224411 10283 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 15:07:29.224534 10283 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:07:29.225518 10256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.206265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.4.205.254:33261)
0925 15:07:29.206458 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.206465 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.206998 (+ 533us) server_negotiation.cc:407] Connection header received
0925 15:07:29.207191 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.207195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.207259 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.207398 (+ 139us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:07:29.207932 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.208642 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.210651 (+ 2009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.211059 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.211148 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.211657 (+ 509us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:07:29.225015 (+ 13358us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:07:29.225281 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:07:29.225790 10256 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 15:07:29.225921 10282 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:29.206165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33261 (local address 127.0.0.1:54408)
0925 15:07:29.206880 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.206910 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.207051 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.207441 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.207446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.207459 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:07:29.207772 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.207783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.208767 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.208790 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.210524 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.210539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.211213 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.211219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.211403 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.211447 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:07:29.225229 (+ 13782us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 15:07:29.225487 (+ 258us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.205.254:33261: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":567,"thread_start_us":421,"threads_started":1}
I20260925 15:07:29.227049 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10187
I20260925 15:07:29.228260 10248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:29.318293 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10187
2026-09-25T15:07:29Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (696 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 15:07:29.372707 4919 mini_oidc.cc:152] Starting JWKS server
I20260925 15:07:29.375167 4919 webserver.cc:460] Webserver started at https://127.0.0.1:42385/ using document root <none> and password file <none>
I20260925 15:07:29.375413 4919 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 15:07:29.376027 4919 webserver.cc:460] Webserver started at http://127.0.0.1:40219/ using document root <none> and password file <none>
2026-09-25T15:07:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:29Z Disabled control of system clock
I20260925 15:07:29.400820 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:36439
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:43299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:36439
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42385/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:07:29.701454 10290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:29.701723 10290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:29.701804 10290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:29.708707 10290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:29.708811 10290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:29.708848 10290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:29.708894 10290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:29.708930 10290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:07:29.708988 10290 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:07:29.709028 10290 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42385/jwks.json
I20260925 15:07:29.717746 10290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:36439
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:36439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42385/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:29.718847 10290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:29.720453 10290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:29.731604 10296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.731644 10295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:29.731669 10298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:29.732280 10290 server_base.cc:1034] running on GCE node
I20260925 15:07:29.732986 10290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:29.734344 10290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:29.735538 10290 hybrid_clock.cc:648] HybridClock initialized: now 1790348849735493 us; error 40 us; skew 500 ppm
I20260925 15:07:29.735867 10290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:29.742309 10290 webserver.cc:460] Webserver started at http://127.4.205.254:36959/ using document root <none> and password file <none>
I20260925 15:07:29.742969 10290 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:29.743078 10290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:29.743394 10290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:29.746224 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "32aa35373c244fc6888a6f39ef894847"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "4ab9969afab926c3421f072c27030fe3"
server_key_iv: "3fd789de595818e77483c833403506bb"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.746984 10290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "32aa35373c244fc6888a6f39ef894847"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "4ab9969afab926c3421f072c27030fe3"
server_key_iv: "3fd789de595818e77483c833403506bb"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.751773 10290 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:29.754863 10304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:29.756042 10290 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:29.756211 10290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "32aa35373c244fc6888a6f39ef894847"
format_stamp: "Formatted at 2026-09-25 15:07:29 on dist-test-slave-64fx"
server_key: "4ab9969afab926c3421f072c27030fe3"
server_key_iv: "3fd789de595818e77483c833403506bb"
server_key_version: "encryptionkey@0"
I20260925 15:07:29.756443 10290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:29.774297 10290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:29.775123 10290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:29.775408 10290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:29.792794 10290 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:36439
I20260925 15:07:29.792812 10355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:36439 every 8 connection(s)
I20260925 15:07:29.794081 10290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:29.795742 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10290
I20260925 15:07:29.795946 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:29.796696 4919 external_mini_cluster.cc:1428] Setting key 6093bcb0d0930ce968352d060d2925c9
I20260925 15:07:29.798192 10356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:29.809013 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.798546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58818 (local address 127.4.205.254:36439)
0925 15:07:29.799405 (+ 859us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.799417 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.799453 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:07:29.800879 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.800898 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.801353 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.801765 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:29.802308 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.803562 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.804946 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.805640 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.805818 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.806961 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:29.806968 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:29.806983 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:29.807489 (+ 506us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:29.807555 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:29.807582 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:29.807703 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:29.807924 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:07:29.808180 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":342,"threads_started":1}
I20260925 15:07:29.809098 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847: Bootstrap starting.
I20260925 15:07:29.812064 10356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:29.812394 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:29.813004 10356 log.cc:824] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:29.815573 10356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847: No bootstrap required, opened a new log
I20260925 15:07:29.822456 10356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } }
I20260925 15:07:29.822831 10356 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:29.822918 10356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32aa35373c244fc6888a6f39ef894847, State: Initialized, Role: FOLLOWER
I20260925 15:07:29.823446 10356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } }
I20260925 15:07:29.823648 10356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:29.823779 10356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:29.823958 10356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:29.826061 10356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } }
I20260925 15:07:29.826494 10356 leader_election.cc:302] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32aa35373c244fc6888a6f39ef894847; no voters:
I20260925 15:07:29.827028 10356 leader_election.cc:288] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:29.827160 10361 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:29.827821 10361 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [term 1 LEADER]: Becoming Leader. State: Replica: 32aa35373c244fc6888a6f39ef894847, State: Running, Role: LEADER
I20260925 15:07:29.828323 10361 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } }
I20260925 15:07:29.828938 10356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:29.830986 10363 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32aa35373c244fc6888a6f39ef894847. Latest consensus state: current_term: 1 leader_uuid: "32aa35373c244fc6888a6f39ef894847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } } }
I20260925 15:07:29.831333 10363 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:29.831105 10362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32aa35373c244fc6888a6f39ef894847" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32aa35373c244fc6888a6f39ef894847" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36439 } } }
I20260925 15:07:29.831491 10362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:29.833662 10368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:29.838263 10368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:29.845669 10368 catalog_manager.cc:1347] Generated new cluster ID: 082647bd98bc46f884bb8efc1d16e548
I20260925 15:07:29.845763 10368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:29.872053 10368 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:29.873891 10368 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:29.885689 10368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32aa35373c244fc6888a6f39ef894847: Generated new TSK 0
I20260925 15:07:29.886620 10368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:29.914091 10359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:29.906321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.4.205.254:36439)
0925 15:07:29.906520 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:29.906526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:29.906921 (+ 395us) server_negotiation.cc:407] Connection header received
0925 15:07:29.907114 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:29.907120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:29.907210 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:29.907366 (+ 156us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:07:29.907743 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.908892 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.910175 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.910855 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.910983 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.911389 (+ 406us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:07:29.913507 (+ 2118us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:07:29.913819 (+ 312us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 15:07:29.914201 10359 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58824: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 15:07:29.915814 10385 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:29.906195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:36439 (local address 127.0.0.1:58824)
0925 15:07:29.906837 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:29.906851 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:29.906973 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:29.907373 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:29.907376 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:29.907385 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:07:29.907616 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.907623 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.909015 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.909019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.910054 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:29.910064 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:29.911008 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:29.911012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:29.911121 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:29.911148 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:07:29.913704 (+ 2556us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 15:07:29.915442 (+ 1738us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.205.254:36439: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42385/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":1472,"thread_start_us":362,"threads_started":1}
I20260925 15:07:29.916873 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10290
I20260925 15:07:29.917932 10351 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 15:07:29.925819 10386 webserver.cc:533] Webserver: sslize failed:
E20260925 15:07:29.925892 10386 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 15:07:30.004714 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10290
2026-09-25T15:07:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (686 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T15:07:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:30Z Disabled control of system clock
I20260925 15:07:30.081527 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37217
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:37217
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/testchainca.cert with env {}
W20260925 15:07:30.387068 10391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:30.387320 10391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:30.387399 10391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:30.393829 10391 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:30.393914 10391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:30.393950 10391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:30.394001 10391 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:30.394061 10391 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 15:07:30.394151 10391 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 15:07:30.394209 10391 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 15:07:30.406141 10391 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:37217
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:30.407253 10391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:30.408455 10391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:30.419668 10397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:30.419632 10399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:30.419723 10396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:30.422866 10391 server_base.cc:1034] running on GCE node
I20260925 15:07:30.423568 10391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:30.424728 10391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:30.425912 10391 hybrid_clock.cc:648] HybridClock initialized: now 1790348850425884 us; error 26 us; skew 500 ppm
I20260925 15:07:30.426186 10391 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:07:30.428104 10391 webserver.cc:460] Webserver started at http://127.4.205.254:40913/ using document root <none> and password file <none>
I20260925 15:07:30.428678 10391 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:30.428768 10391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:30.429023 10391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:30.431576 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "63399472bba94b5fbb51230c2e5ce6e5"
format_stamp: "Formatted at 2026-09-25 15:07:30 on dist-test-slave-64fx"
server_key: "349c78aad53430569176f0deb6b92f39"
server_key_iv: "935c038e0be5032bceb6f4a8dd67b424"
server_key_version: "encryptionkey@0"
I20260925 15:07:30.432248 10391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "63399472bba94b5fbb51230c2e5ce6e5"
format_stamp: "Formatted at 2026-09-25 15:07:30 on dist-test-slave-64fx"
server_key: "349c78aad53430569176f0deb6b92f39"
server_key_iv: "935c038e0be5032bceb6f4a8dd67b424"
server_key_version: "encryptionkey@0"
I20260925 15:07:30.436730 10391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:07:30.439854 10405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:30.441030 10391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:30.441159 10391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "63399472bba94b5fbb51230c2e5ce6e5"
format_stamp: "Formatted at 2026-09-25 15:07:30 on dist-test-slave-64fx"
server_key: "349c78aad53430569176f0deb6b92f39"
server_key_iv: "935c038e0be5032bceb6f4a8dd67b424"
server_key_version: "encryptionkey@0"
I20260925 15:07:30.441337 10391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:30.487344 10391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:30.488018 10391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:30.488233 10391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:30.505281 10391 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:37217
I20260925 15:07:30.505300 10456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:37217 every 8 connection(s)
I20260925 15:07:30.506479 10391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:30.510021 10457 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:30.516853 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10391
I20260925 15:07:30.517059 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:30.517674 4919 external_mini_cluster.cc:1428] Setting key 1eb65280ff1e1a7cbb5cdaf49c930513
I20260925 15:07:30.519411 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5: Bootstrap starting.
I20260925 15:07:30.522315 10457 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:30.523530 10457 log.cc:824] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:30.526328 10457 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5: No bootstrap required, opened a new log
I20260925 15:07:30.530408 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.519335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52520 (local address 127.4.205.254:37217)
0925 15:07:30.520244 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.520253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.520288 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:07:30.521917 (+ 1629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.521937 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.522323 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.522673 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:30.523242 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.524849 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.526613 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.527250 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.527375 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.528475 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:30.528482 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:30.528495 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:07:30.528952 (+ 457us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:30.529018 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:07:30.529034 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:30.529110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:30.529314 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:07:30.529608 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":378,"threads_started":1}
I20260925 15:07:30.533653 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:07:30.534164 10457 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } }
I20260925 15:07:30.534468 10457 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:30.534574 10457 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63399472bba94b5fbb51230c2e5ce6e5, State: Initialized, Role: FOLLOWER
I20260925 15:07:30.535116 10457 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } }
I20260925 15:07:30.535342 10457 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:30.535485 10457 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:30.535640 10457 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:30.537915 10457 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } }
I20260925 15:07:30.538322 10457 leader_election.cc:302] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63399472bba94b5fbb51230c2e5ce6e5; no voters:
I20260925 15:07:30.538727 10457 leader_election.cc:288] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:30.538864 10462 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:30.539481 10462 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [term 1 LEADER]: Becoming Leader. State: Replica: 63399472bba94b5fbb51230c2e5ce6e5, State: Running, Role: LEADER
I20260925 15:07:30.539987 10462 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } }
I20260925 15:07:30.540421 10457 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:30.543774 10464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63399472bba94b5fbb51230c2e5ce6e5. Latest consensus state: current_term: 1 leader_uuid: "63399472bba94b5fbb51230c2e5ce6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } } }
I20260925 15:07:30.543798 10463 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63399472bba94b5fbb51230c2e5ce6e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63399472bba94b5fbb51230c2e5ce6e5" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37217 } } }
I20260925 15:07:30.544250 10464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:30.544268 10463 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:30.546983 10473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:30.551323 10473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:30.558599 10473 catalog_manager.cc:1347] Generated new cluster ID: 35165835a2784d0cb0df0d275b4f9845
I20260925 15:07:30.558692 10473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:30.571914 10473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:30.573030 10473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:30.583776 10473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63399472bba94b5fbb51230c2e5ce6e5: Generated new TSK 0
I20260925 15:07:30.584649 10473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:30.731081 10460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:30.602001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.4.205.254:37217)
0925 15:07:30.602217 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:30.602223 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:30.602565 (+ 342us) server_negotiation.cc:407] Connection header received
0925 15:07:30.602770 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:30.602773 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:30.602836 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:30.602972 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0925 15:07:30.603470 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.604165 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.606176 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.606594 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.606686 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.607235 (+ 549us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 15:07:30.730527 (+123292us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:07:30.730845 (+ 318us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:07:30.731228 10460 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52526: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 15:07:30.731812 10486 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:30.601890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37217 (local address 127.0.0.1:52526)
0925 15:07:30.602466 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:30.602485 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:30.602618 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:30.603005 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:30.603011 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:30.603026 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0925 15:07:30.603330 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.603340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.604292 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.604323 (+ 31us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.606050 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:30.606065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:30.606746 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:30.606752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:30.606929 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:30.606970 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 15:07:30.730821 (+123851us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 15:07:30.731103 (+ 282us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.4.205.254:37217: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":290,"threads_started":1}
I20260925 15:07:30.732961 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10391
I20260925 15:07:30.734042 10452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:30.830307 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10391
2026-09-25T15:07:30Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (805 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (389 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (393 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:31 dist-test-slave-64fx krb5kdc[10501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:31 dist-test-slave-64fx krb5kdc[10501](info): set up 2 sockets
Sep 25 15:07:31 dist-test-slave-64fx krb5kdc[10501](info): commencing operation
krb5kdc: starting...
W20260925 15:07:32.674206 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:32 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:32.865429 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:36181
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:36181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:33.168237 10514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:33.168537 10514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:33.168689 10514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:33.175796 10514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:33.175891 10514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:33.175977 10514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:33.176076 10514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:33.176163 10514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:33.186064 10514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:36181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:36181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:33.187300 10514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:33.188565 10514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:33.199283 10520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:33.199324 10522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:33.199349 10519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:33.200441 10514 server_base.cc:1034] running on GCE node
I20260925 15:07:33.201050 10514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:33.202345 10514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:33.203541 10514 hybrid_clock.cc:648] HybridClock initialized: now 1790348853203503 us; error 26 us; skew 500 ppm
I20260925 15:07:33.203800 10514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:33 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:33.206667 10514 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:33.208066 10514 webserver.cc:460] Webserver started at http://127.4.205.254:41171/ using document root <none> and password file <none>
I20260925 15:07:33.208684 10514 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:33.208807 10514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:33.209108 10514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:33.212194 10514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "3a016b1a68c94ea59b252878fc8dfbcf"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "6a60aea520e6e38808a6bc4168dd121e"
server_key_iv: "37efd8d17ccff79067e5caaa0425540a"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.212989 10514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "3a016b1a68c94ea59b252878fc8dfbcf"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "6a60aea520e6e38808a6bc4168dd121e"
server_key_iv: "37efd8d17ccff79067e5caaa0425540a"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.218827 10514 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 15:07:33.221946 10529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:33.223174 10514 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:07:33.223368 10514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "3a016b1a68c94ea59b252878fc8dfbcf"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "6a60aea520e6e38808a6bc4168dd121e"
server_key_iv: "37efd8d17ccff79067e5caaa0425540a"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.223560 10514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:33.247511 10514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:33.251039 10514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:33.251309 10514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:33.268882 10514 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:36181
I20260925 15:07:33.268915 10580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:36181 every 8 connection(s)
I20260925 15:07:33.270267 10514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:33.271438 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10514
I20260925 15:07:33.271639 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:33.272326 4919 external_mini_cluster.cc:1428] Setting key 404a848f0accc9a2228c966b42f73834
I20260925 15:07:33.274489 10581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:07:33 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:33.287029 10581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf: Bootstrap starting.
I20260925 15:07:33.289975 10581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:33.290918 10581 log.cc:824] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:33.293550 10581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf: No bootstrap required, opened a new log
I20260925 15:07:33.294149 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.274196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.4.205.254:36181)
0925 15:07:33.275009 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:33.275020 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:33.275060 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:07:33.276643 (+ 1583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:33.276662 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:33.277012 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:33.277332 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:33.279002 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.280269 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.281991 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.282604 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.282742 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:33.287098 (+ 4356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:33.287105 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:33.287118 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:33.287178 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:33.290227 (+ 3049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:33.290882 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:33.290889 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:33.290894 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:33.291091 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:33.291472 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:33.291478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:33.291483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:33.292001 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:33.292314 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:33.292706 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:07:33.293097 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":404,"threads_started":1}
I20260925 15:07:33.301105 10581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } }
I20260925 15:07:33.301501 10581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:33.301618 10581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a016b1a68c94ea59b252878fc8dfbcf, State: Initialized, Role: FOLLOWER
I20260925 15:07:33.302173 10581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } }
I20260925 15:07:33.302395 10581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:33.302526 10581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:33.302695 10581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:33.304942 10581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } }
I20260925 15:07:33.305372 10581 leader_election.cc:302] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a016b1a68c94ea59b252878fc8dfbcf; no voters:
I20260925 15:07:33.305934 10581 leader_election.cc:288] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:33.306051 10586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:33.306707 10586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [term 1 LEADER]: Becoming Leader. State: Replica: 3a016b1a68c94ea59b252878fc8dfbcf, State: Running, Role: LEADER
I20260925 15:07:33.307191 10586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } }
I20260925 15:07:33.308073 10581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:33.310065 10588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a016b1a68c94ea59b252878fc8dfbcf. Latest consensus state: current_term: 1 leader_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } } }
I20260925 15:07:33.310468 10588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:33.311009 10587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a016b1a68c94ea59b252878fc8dfbcf" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 36181 } } }
I20260925 15:07:33.311381 10587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:33.312319 10593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:33.316282 10593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:33.324726 10593 catalog_manager.cc:1347] Generated new cluster ID: 429b2748acb4457f99ca9b2dd581bd39
I20260925 15:07:33.324833 10593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:33.348888 10593 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:33.350154 10593 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:33.360275 10593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a016b1a68c94ea59b252878fc8dfbcf: Generated new TSK 0
I20260925 15:07:33.361004 10593 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:33.409314 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:33.721251 10607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:33.721498 10607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:33.721647 10607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:33.728597 10607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:33.728681 10607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:33.728786 10607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:33.737370 10607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:33.738453 10607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:33.739605 10607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:33.751520 10613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:33.751667 10612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:33.751606 10615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:33.752497 10607 server_base.cc:1034] running on GCE node
I20260925 15:07:33.753034 10607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:33.753963 10607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:33.755172 10607 hybrid_clock.cc:648] HybridClock initialized: now 1790348853755132 us; error 49 us; skew 500 ppm
I20260925 15:07:33.755462 10607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:33 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:33.758347 10607 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:33.759575 10607 webserver.cc:460] Webserver started at http://127.4.205.193:34707/ using document root <none> and password file <none>
I20260925 15:07:33.760154 10607 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:33.760244 10607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:33.760473 10607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:33.763239 10607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "a36ccf35509243d0ba81bb07878bf3ec"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "e380f10d5777870f2e6b932b9739f4e0"
server_key_iv: "b54aa753f445005aecb3ab45a0c740b0"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.763923 10607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "a36ccf35509243d0ba81bb07878bf3ec"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "e380f10d5777870f2e6b932b9739f4e0"
server_key_iv: "b54aa753f445005aecb3ab45a0c740b0"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.768463 10607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:07:33.771587 10622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:33.772773 10607 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:33.772893 10607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "a36ccf35509243d0ba81bb07878bf3ec"
format_stamp: "Formatted at 2026-09-25 15:07:33 on dist-test-slave-64fx"
server_key: "e380f10d5777870f2e6b932b9739f4e0"
server_key_iv: "b54aa753f445005aecb3ab45a0c740b0"
server_key_version: "encryptionkey@0"
I20260925 15:07:33.773054 10607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:33.799368 10607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:33.803056 10607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:33.803283 10607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:33.804091 10607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:33.805207 10607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:33.805270 10607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:33.805342 10607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:33.805380 10607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:33.836862 10607 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:45703
I20260925 15:07:33.836998 10734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:45703 every 8 connection(s)
I20260925 15:07:33.838151 10607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:33.847254 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10607
I20260925 15:07:33.847474 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:33.848157 4919 external_mini_cluster.cc:1428] Setting key c9aadb277d5dad250441b901bd13deca
Sep 25 15:07:33 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348853, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:33.856806 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.840468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:51425 (local address 127.4.205.254:36181)
0925 15:07:33.840869 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:33.840873 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:33.841684 (+ 811us) server_negotiation.cc:407] Connection header received
0925 15:07:33.843403 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:33.843408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:33.843491 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:33.843603 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:33.844940 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.845745 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.847505 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.847994 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.848093 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:33.851706 (+ 3613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:33.851711 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:33.851714 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:33.851741 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:33.854093 (+ 2352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:33.854760 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:33.854773 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:33.854776 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:33.854879 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:33.855274 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:33.855281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:33.855286 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:33.855541 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:33.855683 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:33.856262 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 15:07:33.856460 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":223,"threads_started":1}
I20260925 15:07:33.857749 10738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:33.840833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:36181 (local address 127.4.205.193:51425)
0925 15:07:33.841496 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:33.841530 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:33.843160 (+ 1630us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:33.843818 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:33.843825 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:33.844280 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:33.844778 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:33.844793 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.845921 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.845929 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:33.847368 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:33.847385 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:33.848208 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:33.848217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:33.848492 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:33.849205 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:33.849234 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:33.851500 (+ 2266us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:33.854229 (+ 2729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:33.854238 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:33.854251 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:33.854631 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:33.854988 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:33.854994 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:33.854999 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:33.855168 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:33.855657 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:33.855664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:33.855946 (+ 282us) client_negotiation.cc:769] Sending connection context
0925 15:07:33.856188 (+ 242us) client_negotiation.cc:240] Negotiation successful
0925 15:07:33.856639 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":259,"threads_started":1}
I20260925 15:07:33.859120 10735 heartbeater.cc:347] Connected to a master server at 127.4.205.254:36181
I20260925 15:07:33.859405 10735 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:33.860154 10735 heartbeater.cc:511] Master 127.4.205.254:36181 requested a full tablet report, sending...
I20260925 15:07:33.862387 10546 ts_manager.cc:194] Registered new tserver with Master: a36ccf35509243d0ba81bb07878bf3ec (127.4.205.193:45703)
I20260925 15:07:33.864468 10546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:51425
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:33.890895 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:34.189296 10741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:34.189574 10741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:34.189713 10741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:34.197029 10741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:34.197115 10741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:34.197212 10741 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:34.206073 10741 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:34.207446 10741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:34.208758 10741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:34.222400 10747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:34.222586 10749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:34.223212 10746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:34.223668 10741 server_base.cc:1034] running on GCE node
I20260925 15:07:34.224288 10741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:34.225147 10741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:34.226370 10741 hybrid_clock.cc:648] HybridClock initialized: now 1790348854226300 us; error 59 us; skew 500 ppm
I20260925 15:07:34.226707 10741 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.229635 10741 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:34.230934 10741 webserver.cc:460] Webserver started at http://127.4.205.194:37357/ using document root <none> and password file <none>
I20260925 15:07:34.231499 10741 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:34.231590 10741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:34.231817 10741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:34.234503 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "94dc0bc23ca546159964a80859a973f4"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "685f3f87b14206a9eaab0749433dc9e1"
server_key_iv: "6638f2317ad6c49424980f9b5722b3e0"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.235196 10741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "94dc0bc23ca546159964a80859a973f4"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "685f3f87b14206a9eaab0749433dc9e1"
server_key_iv: "6638f2317ad6c49424980f9b5722b3e0"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.239799 10741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:34.242933 10756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.244098 10741 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:34.244256 10741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "94dc0bc23ca546159964a80859a973f4"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "685f3f87b14206a9eaab0749433dc9e1"
server_key_iv: "6638f2317ad6c49424980f9b5722b3e0"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.244458 10741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:34.259814 10741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:34.263427 10741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:34.263732 10741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:34.264604 10741 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:34.265895 10741 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:34.265973 10741 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.266131 10741 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:34.266183 10741 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.299361 10741 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:39631
I20260925 15:07:34.299448 10868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:39631 every 8 connection(s)
I20260925 15:07:34.300611 10741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:34.306383 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10741
I20260925 15:07:34.306588 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:34.307013 4919 external_mini_cluster.cc:1428] Setting key 427515ad9b682c83c0812d636917e3cb
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:34.320704 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.303087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:35693 (local address 127.4.205.254:36181)
0925 15:07:34.303256 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.303260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.304390 (+ 1130us) server_negotiation.cc:407] Connection header received
0925 15:07:34.306180 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.306185 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.306282 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.306401 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.308089 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.309089 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.310656 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.311415 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.311552 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.314705 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.314713 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.314718 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.314761 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.317551 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.318473 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.318477 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.318479 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.318592 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.319048 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.319051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.319053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.319258 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.319362 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.320138 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.320365 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:07:34.321951 10871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.303552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:36181 (local address 127.4.205.194:35693)
0925 15:07:34.304225 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:34.304259 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:34.305939 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:34.306641 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:34.306649 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:34.307227 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:34.307878 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:34.307897 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.309234 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.309237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:34.310518 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:34.310529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.311612 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.311616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:34.311786 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.312434 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:34.312454 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:34.314514 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:34.317724 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:34.317735 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:34.317752 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:34.318361 (+ 609us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:34.318702 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:34.318710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:34.318716 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:34.318947 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:34.319375 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:34.319386 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:34.319784 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 15:07:34.320060 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 15:07:34.320551 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":259,"threads_started":1}
I20260925 15:07:34.323294 10869 heartbeater.cc:347] Connected to a master server at 127.4.205.254:36181
I20260925 15:07:34.323624 10869 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:34.324323 10869 heartbeater.cc:511] Master 127.4.205.254:36181 requested a full tablet report, sending...
I20260925 15:07:34.325759 10546 ts_manager.cc:194] Registered new tserver with Master: 94dc0bc23ca546159964a80859a973f4 (127.4.205.194:39631)
I20260925 15:07:34.326535 10546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:35693
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:34.352672 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:32965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:34.663166 10874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:34.663450 10874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:34.663631 10874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:34.670357 10874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:34.670495 10874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:34.670675 10874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:34.680588 10874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:32965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:36181
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:34.681829 10874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:34.683105 10874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:34.695182 10879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:34.695294 10880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:34.695329 10882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:34.696202 10874 server_base.cc:1034] running on GCE node
I20260925 15:07:34.696777 10874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:34.697667 10874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:34.698882 10874 hybrid_clock.cc:648] HybridClock initialized: now 1790348854698825 us; error 45 us; skew 500 ppm
I20260925 15:07:34.699185 10874 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.702129 10874 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:34.703372 10874 webserver.cc:460] Webserver started at http://127.4.205.195:41559/ using document root <none> and password file <none>
I20260925 15:07:34.703962 10874 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:34.704072 10874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:34.704298 10874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:34.706948 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "ad1ee545e5c04a6d802ed2a0da91985a"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "cc3904cce4c60ba1df8d2494ca210790"
server_key_iv: "ce1f47354b2fcb876458b27a073f579b"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.707623 10874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "ad1ee545e5c04a6d802ed2a0da91985a"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "cc3904cce4c60ba1df8d2494ca210790"
server_key_iv: "ce1f47354b2fcb876458b27a073f579b"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.712319 10874 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:07:34.715610 10889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.716799 10874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:34.716919 10874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "ad1ee545e5c04a6d802ed2a0da91985a"
format_stamp: "Formatted at 2026-09-25 15:07:34 on dist-test-slave-64fx"
server_key: "cc3904cce4c60ba1df8d2494ca210790"
server_key_iv: "ce1f47354b2fcb876458b27a073f579b"
server_key_version: "encryptionkey@0"
I20260925 15:07:34.717056 10874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:34.760250 10874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:34.763968 10874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:34.764215 10874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:34.765067 10874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:34.766340 10874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:34.766407 10874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.766482 10874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:34.766525 10874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:34.797865 10874 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:42731
I20260925 15:07:34.797885 11001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:42731 every 8 connection(s)
I20260925 15:07:34.799088 10874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:34.806097 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 10874
I20260925 15:07:34.806263 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:34.806783 4919 external_mini_cluster.cc:1428] Setting key e6132ee6ceec218bf5a70ebee00b2dba
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:34.818338 10737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.801488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:55675 (local address 127.4.205.254:36181)
0925 15:07:34.801718 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.801726 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.803036 (+ 1310us) server_negotiation.cc:407] Connection header received
0925 15:07:34.804745 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.804751 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.804838 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.804940 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.806629 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.807572 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.809528 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.810020 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.810127 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.813112 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.813117 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.813120 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.813149 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.815752 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.816374 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.816378 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.816398 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.816517 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.816959 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.816962 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.816964 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.817196 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.817295 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.817931 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.818086 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:07:34.819234 11004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.802200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:36181 (local address 127.4.205.195:55675)
0925 15:07:34.802894 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:34.802933 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:34.804474 (+ 1541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:34.805151 (+ 677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:34.805158 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:34.805682 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:34.806465 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:34.806489 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.807725 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.807729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:34.809398 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:34.809409 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.810203 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.810207 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:34.810396 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.810970 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:34.810989 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:34.812951 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:34.815882 (+ 2931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:34.815888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:34.815898 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:34.816275 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:34.816612 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:34.816614 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:34.816616 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:34.816863 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:34.817300 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:34.817307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:34.817635 (+ 328us) client_negotiation.cc:769] Sending connection context
0925 15:07:34.817862 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 15:07:34.818192 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":283,"threads_started":1}
I20260925 15:07:34.820554 11002 heartbeater.cc:347] Connected to a master server at 127.4.205.254:36181
I20260925 15:07:34.820856 11002 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:34.821779 11002 heartbeater.cc:511] Master 127.4.205.254:36181 requested a full tablet report, sending...
I20260925 15:07:34.823061 10546 ts_manager.cc:194] Registered new tserver with Master: ad1ee545e5c04a6d802ed2a0da91985a (127.4.205.195:42731)
I20260925 15:07:34.823815 10546 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:55675
I20260925 15:07:34.833691 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/test-admin.keytab.
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.858858 4919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 15:07:34.863605 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:34.861391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39948 (local address 127.4.205.193:45703)
0925 15:07:34.862012 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.862020 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.862045 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:34.862102 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.862108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.862310 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.862435 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.863239 (+ 804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39948: BlockingRecv error: recv got EOF from 127.0.0.1:39948 (error 108)
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":353,"threads_started":1}
W20260925 15:07:34.863930 4919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.865393 4919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.866750 4919 init.cc:295] Successfully reacquired a new kerberos TGT
I20260925 15:07:34.867936 10735 heartbeater.cc:503] Master 127.4.205.254:36181 was elected leader, sending a full tablet report...
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:34.881124 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.868450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.4.205.193:45703)
0925 15:07:34.868629 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.868637 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.868937 (+ 300us) server_negotiation.cc:407] Connection header received
0925 15:07:34.869104 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.869108 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.869160 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.869235 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.870341 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.871246 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.872378 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.872892 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.872983 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.876222 (+ 3239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.876231 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.876244 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.876270 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.878852 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.879441 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.879447 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.879449 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.879549 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.879849 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.879852 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.879854 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.880151 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.880343 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.880595 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.880863 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260925 15:07:34.886180 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:34.884654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39972 (local address 127.4.205.193:45703)
0925 15:07:34.884807 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.884813 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.885122 (+ 309us) server_negotiation.cc:407] Connection header received
0925 15:07:34.885223 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.885226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.885271 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.885348 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.885942 (+ 594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39972: BlockingRecv error: recv got EOF from 127.0.0.1:39972 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 15:07:34.886394 4919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.887784 4919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.889007 4919 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:34.902518 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.890222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.4.205.193:45703)
0925 15:07:34.890403 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.890411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.890732 (+ 321us) server_negotiation.cc:407] Connection header received
0925 15:07:34.890902 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.890908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.890979 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.891073 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.891941 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.892627 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.893865 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.894299 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.894428 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.897868 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.897875 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.897881 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.897937 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.900088 (+ 2151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.900682 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.900688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.900693 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.900816 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.901198 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.901204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.901208 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.901483 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.901759 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.902063 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.902251 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 15:07:34.906237 11009 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:34.904774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39984 (local address 127.4.205.193:45703)
0925 15:07:34.904960 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.904965 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.905164 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:07:34.905265 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.905269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.905327 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.905398 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.906037 (+ 639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39984: BlockingRecv error: recv got EOF from 127.0.0.1:39984 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260925 15:07:34.906374 4919 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348808911781-4919-0/krb5kdc/krb5cc' of type FILE
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.907842 4919 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:34.909090 4919 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 15:07:34 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:34.921983 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.910393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.4.205.193:45703)
0925 15:07:34.910560 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.910567 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.910731 (+ 164us) server_negotiation.cc:407] Connection header received
0925 15:07:34.910832 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.910835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.910882 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.910955 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.911794 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.912457 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.913601 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.914011 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.914100 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.917346 (+ 3246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.917350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.917353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.917379 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.919698 (+ 2319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.920263 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.920267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.920269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.920401 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.920737 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.920740 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.920742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.920967 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.921174 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.921407 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.921604 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:07:34.923408 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10607
I20260925 15:07:34.934023 11009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:34.924054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40006 (local address 127.4.205.193:45703)
0925 15:07:34.924223 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:34.924230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:34.924511 (+ 281us) server_negotiation.cc:407] Connection header received
0925 15:07:34.924609 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:34.924612 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:34.924660 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:34.924729 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:34.925505 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.926181 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.927358 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:34.927832 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:34.927934 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:34.929738 (+ 1804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:34.929742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:34.929745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:34.929768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:34.931905 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.932440 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.932443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.932445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.932560 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:34.932871 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:34.932874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:34.932876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:34.933083 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:34.933251 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:34.933483 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:07:34.933712 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:07:34.934653 10730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:35.015100 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10607
I20260925 15:07:35.031262 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10741
Sep 25 15:07:35 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:35.046027 11022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:35.032060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56254 (local address 127.4.205.194:39631)
0925 15:07:35.032607 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:35.032614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:35.032637 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:35.032692 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:35.032698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:35.032906 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:35.033071 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:35.034130 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:35.035136 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:35.036346 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:35.036886 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:35.037019 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:35.040675 (+ 3656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:35.040685 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:35.040698 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:35.040729 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:35.043341 (+ 2612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:35.044017 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:35.044024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:35.044026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:35.044137 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:35.044509 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:35.044516 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:35.044521 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:35.044855 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:35.045160 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:35.045454 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:07:35.045731 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":280,"threads_started":1}
I20260925 15:07:35.047288 10864 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:35.128919 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10741
I20260925 15:07:35.143989 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10874
Sep 25 15:07:35 dist-test-slave-64fx krb5kdc[10501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:35.159061 11024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:35.144890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.4.205.195:42731)
0925 15:07:35.145476 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:35.145483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:35.145511 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:07:35.145618 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:35.145632 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:35.145842 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:35.145973 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:35.146942 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:35.147989 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:35.149181 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:35.149881 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:35.149989 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:35.153524 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:35.153534 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:35.153548 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:35.153654 (+ 106us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:35.156528 (+ 2874us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:35.157140 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:35.157147 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:35.157149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:35.157260 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:35.157752 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:35.157757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:35.157759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:35.158087 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:35.158311 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:35.158499 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 15:07:35.158748 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":306,"threads_started":1}
I20260925 15:07:35.160306 10997 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:35.242259 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10874
I20260925 15:07:35.257427 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 10514
I20260925 15:07:35.258566 10576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:35.354351 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 10514
2026-09-25T15:07:35Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3742 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:35 dist-test-slave-64fx krb5kdc[11029](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:35 dist-test-slave-64fx krb5kdc[11029](info): set up 2 sockets
Sep 25 15:07:35 dist-test-slave-64fx krb5kdc[11029](info): commencing operation
krb5kdc: starting...
W20260925 15:07:36.426049 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:36 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab.
Entry for principal oryx/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab.
I20260925 15:07:36.608978 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37321
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab
--principal=oryx/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:37321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:36.907775 11042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:36.908025 11042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:36.908181 11042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:36.915011 11042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:36.915115 11042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:36.915159 11042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:36.915237 11042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:36.915300 11042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:36.924746 11042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:37321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab
--principal=oryx/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:37321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:36.925966 11042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:36.927218 11042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:36.938124 11048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:36.938052 11047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:36.938601 11050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:36.939009 11042 server_base.cc:1034] running on GCE node
I20260925 15:07:36.939726 11042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:36.940999 11042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:36.942225 11042 hybrid_clock.cc:648] HybridClock initialized: now 1790348856942185 us; error 51 us; skew 500 ppm
I20260925 15:07:36.942615 11042 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:36 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:36.945602 11042 init.cc:377] Logged in from keytab as oryx/127.4.205.254@KRBTEST.COM (short username oryx)
I20260925 15:07:36.946920 11042 webserver.cc:460] Webserver started at http://127.4.205.254:32957/ using document root <none> and password file <none>
I20260925 15:07:36.947531 11042 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:36.947691 11042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:36.947973 11042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:36.950758 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "fb955e793d8d4980b96753d75ce7903f"
format_stamp: "Formatted at 2026-09-25 15:07:36 on dist-test-slave-64fx"
server_key: "26de6e144456a4bef4698a8d89260c5f"
server_key_iv: "33a3d10e1f8423c5e40d8ccc4a3114ac"
server_key_version: "encryptionkey@0"
I20260925 15:07:36.951494 11042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "fb955e793d8d4980b96753d75ce7903f"
format_stamp: "Formatted at 2026-09-25 15:07:36 on dist-test-slave-64fx"
server_key: "26de6e144456a4bef4698a8d89260c5f"
server_key_iv: "33a3d10e1f8423c5e40d8ccc4a3114ac"
server_key_version: "encryptionkey@0"
I20260925 15:07:36.956221 11042 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260925 15:07:36.959331 11057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:36.960510 11042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:36.960671 11042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "fb955e793d8d4980b96753d75ce7903f"
format_stamp: "Formatted at 2026-09-25 15:07:36 on dist-test-slave-64fx"
server_key: "26de6e144456a4bef4698a8d89260c5f"
server_key_iv: "33a3d10e1f8423c5e40d8ccc4a3114ac"
server_key_version: "encryptionkey@0"
I20260925 15:07:36.960845 11042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:36.971585 11042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:36.975220 11042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:36.975481 11042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:36.992591 11042 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:37321
I20260925 15:07:36.992628 11108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:37321 every 8 connection(s)
I20260925 15:07:36.993854 11042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:36.997506 11109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:37.003727 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11042
I20260925 15:07:37.003917 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:37.004513 4919 external_mini_cluster.cc:1428] Setting key 0cf4443e6e7c8e94de43a0a7a30c2675
I20260925 15:07:37.007135 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Bootstrap starting.
I20260925 15:07:37.010301 11109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:37.011412 11109 log.cc:824] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:37.014334 11109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: No bootstrap required, opened a new log
Sep 25 15:07:37 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:37.023092 11109 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } }
I20260925 15:07:37.023442 11109 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:37.023573 11109 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb955e793d8d4980b96753d75ce7903f, State: Initialized, Role: FOLLOWER
I20260925 15:07:37.024196 11109 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } }
I20260925 15:07:37.024453 11109 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:37.024593 11109 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:37.024768 11109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:37.026767 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.006244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.4.205.254:37321)
0925 15:07:37.006933 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:37.006942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:37.006980 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:07:37.008566 (+ 1586us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:37.008586 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:37.008937 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:37.009216 (+ 279us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:37.010906 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.012629 (+ 1723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.014250 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.014903 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.015024 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:37.019323 (+ 4299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:37.019330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:37.019342 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:37.019374 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:37.022451 (+ 3077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:37.023149 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:37.023154 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:37.023160 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:37.023316 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:37.023756 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:37.023761 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:37.023764 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:37.024365 (+ 601us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:37.024685 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:37.025081 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 15:07:37.025494 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":324,"threads_started":1}
I20260925 15:07:37.027626 11109 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } }
I20260925 15:07:37.028095 11109 leader_election.cc:302] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb955e793d8d4980b96753d75ce7903f; no voters:
I20260925 15:07:37.028756 11109 leader_election.cc:288] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:37.028873 11114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:37.029943 11114 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 LEADER]: Becoming Leader. State: Replica: fb955e793d8d4980b96753d75ce7903f, State: Running, Role: LEADER
I20260925 15:07:37.030435 11114 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } }
I20260925 15:07:37.030916 11109 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:37.032883 11115 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } }
I20260925 15:07:37.033250 11115 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:37.034147 11121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:37.033983 11116 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb955e793d8d4980b96753d75ce7903f. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } }
I20260925 15:07:37.034394 11116 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:37.040107 11121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:37.048481 11121 catalog_manager.cc:1347] Generated new cluster ID: d168e20da74b422ba6f4e43e61800523
I20260925 15:07:37.048595 11121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:37.061048 11121 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:37.062198 11121 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:37.067948 11121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Generated new TSK 0
I20260925 15:07:37.068655 11121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
I20260925 15:07:37.117252 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:37.433058 11135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:37.433279 11135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:37.433358 11135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:37.439949 11135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:37.440035 11135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:37.440133 11135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:37.448894 11135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:37.450023 11135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:37.451155 11135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:37.463321 11141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:37.463768 11143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:37.463768 11140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:37.464242 11135 server_base.cc:1034] running on GCE node
I20260925 15:07:37.464793 11135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:37.465705 11135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:37.466888 11135 hybrid_clock.cc:648] HybridClock initialized: now 1790348857466855 us; error 30 us; skew 500 ppm
I20260925 15:07:37.467177 11135 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:37 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:37.469976 11135 init.cc:377] Logged in from keytab as oryx/127.4.205.193@KRBTEST.COM (short username oryx)
I20260925 15:07:37.471230 11135 webserver.cc:460] Webserver started at http://127.4.205.193:33265/ using document root <none> and password file <none>
I20260925 15:07:37.471877 11135 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:37.472018 11135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:37.472288 11135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:37.474918 11135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "8987d797c4c14d8f80a376be1d245505"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "fcda735708615dad6dbd6bc8f1213c9c"
server_key_iv: "e361d6031787172cd58a5e362a8d1d9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.475602 11135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "8987d797c4c14d8f80a376be1d245505"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "fcda735708615dad6dbd6bc8f1213c9c"
server_key_iv: "e361d6031787172cd58a5e362a8d1d9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.480198 11135 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:37.483297 11150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:37.484464 11135 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:37.484619 11135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "8987d797c4c14d8f80a376be1d245505"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "fcda735708615dad6dbd6bc8f1213c9c"
server_key_iv: "e361d6031787172cd58a5e362a8d1d9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.484799 11135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:37.504465 11135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:37.507872 11135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:37.508131 11135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:37.508951 11135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:37.510080 11135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:37.510179 11135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:37.510324 11135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:37.510406 11135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:37.541189 11135 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:39305
I20260925 15:07:37.541265 11262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:39305 every 8 connection(s)
I20260925 15:07:37.542512 11135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:37.546103 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11135
I20260925 15:07:37.546288 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:37.546911 4919 external_mini_cluster.cc:1428] Setting key d6f0597d224b7787479741e2db0b16b6
Sep 25 15:07:37 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:37.560789 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.544685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:60515 (local address 127.4.205.254:37321)
0925 15:07:37.545094 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:37.545100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:37.545955 (+ 855us) server_negotiation.cc:407] Connection header received
0925 15:07:37.547913 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:37.547926 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:37.547984 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:37.548074 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:37.549641 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.550346 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.551752 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.552142 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.552268 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:37.555623 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:37.555627 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:37.555630 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:37.555656 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:37.558067 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:37.558726 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:37.558736 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:37.558738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:37.558844 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:37.559253 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:37.559259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:37.559264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:37.559513 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:37.559655 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:37.560235 (+ 580us) server_negotiation.cc:299] Negotiation successful
0925 15:07:37.560463 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":197,"threads_started":1}
I20260925 15:07:37.561731 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:37.545145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:60515)
0925 15:07:37.545820 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:37.545860 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:37.547644 (+ 1784us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:37.548306 (+ 662us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:37.548327 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:37.548973 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:37.549464 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:37.549485 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.550484 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.550496 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:37.551619 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:37.551630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:37.552348 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:37.552357 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:37.552583 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:37.553222 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:37.553242 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:37.555430 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:37.558204 (+ 2774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:37.558213 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:37.558227 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:37.558601 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:37.558963 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:37.558970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:37.558974 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:37.559144 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:37.559630 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:37.559637 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:37.559925 (+ 288us) client_negotiation.cc:769] Sending connection context
0925 15:07:37.560154 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 15:07:37.560601 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":241,"threads_started":1}
I20260925 15:07:37.563158 11263 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:37.563441 11263 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:37.564285 11263 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:07:37.566385 11074 ts_manager.cc:194] Registered new tserver with Master: 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.194@KRBTEST.COM" created.
I20260925 15:07:37.568053 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:60515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
I20260925 15:07:37.589188 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:37.890686 11269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:37.890921 11269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:37.891101 11269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:37.897903 11269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:37.898015 11269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:37.898200 11269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:37.907142 11269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:37.908249 11269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:37.909438 11269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:37.921446 11277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:37.921566 11274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:37.921568 11275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:37.922263 11269 server_base.cc:1034] running on GCE node
I20260925 15:07:37.922818 11269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:37.923637 11269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:37.924846 11269 hybrid_clock.cc:648] HybridClock initialized: now 1790348857924801 us; error 60 us; skew 500 ppm
I20260925 15:07:37.925120 11269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:37 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:37.927942 11269 init.cc:377] Logged in from keytab as oryx/127.4.205.194@KRBTEST.COM (short username oryx)
I20260925 15:07:37.929198 11269 webserver.cc:460] Webserver started at http://127.4.205.194:40033/ using document root <none> and password file <none>
I20260925 15:07:37.929828 11269 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:37.929921 11269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:37.930136 11269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:37.932550 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "0463e3581745fffe395cdbc901b111a8"
server_key_iv: "d641609e5365f5de9246ceddb4c8280a"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.933168 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "0463e3581745fffe395cdbc901b111a8"
server_key_iv: "d641609e5365f5de9246ceddb4c8280a"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.938009 11269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:07:37.940939 11284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:37.942122 11269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:37.942308 11269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "0463e3581745fffe395cdbc901b111a8"
server_key_iv: "d641609e5365f5de9246ceddb4c8280a"
server_key_version: "encryptionkey@0"
I20260925 15:07:37.942505 11269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:37.980577 11269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:37.985955 11269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:37.986266 11269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:37.987464 11269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:37.989373 11269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:37.989526 11269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:07:37.989696 11269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:37.989763 11269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:38.025121 11269 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:33547
I20260925 15:07:38.025153 11396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:33547 every 8 connection(s)
I20260925 15:07:38.026337 11269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:38.032368 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11269
I20260925 15:07:38.032536 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:38.033018 4919 external_mini_cluster.cc:1428] Setting key 2e49c9723d6fd5d41376f1e32b9b3b82
Sep 25 15:07:38 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348857, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:38.045284 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.028530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:60239 (local address 127.4.205.254:37321)
0925 15:07:38.028694 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:38.028699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:38.029757 (+ 1058us) server_negotiation.cc:407] Connection header received
0925 15:07:38.031425 (+ 1668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:38.031429 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:38.031491 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:38.031568 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:38.033262 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.034127 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.036115 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.036575 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.036708 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:38.039958 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:38.039962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:38.039966 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:38.039992 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:38.042534 (+ 2542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:38.043195 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:38.043202 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:38.043208 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:38.043337 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:38.043713 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:38.043720 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:38.043724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:38.044023 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:38.044175 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:38.044764 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 15:07:38.044969 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:07:38.046295 11399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.028920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:60239)
0925 15:07:38.029612 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:38.029656 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:38.031213 (+ 1557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:38.031741 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:38.031748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:38.032403 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:38.033109 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:38.033137 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.034259 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.034264 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:38.035987 (+ 1723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:38.036004 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.036796 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.036803 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:38.037100 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:38.037771 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:38.037794 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:38.039760 (+ 1966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:38.042661 (+ 2901us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:38.042671 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:38.042684 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:38.043054 (+ 370us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:38.043438 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:38.043444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:38.043449 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:38.043610 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:38.044137 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:38.044146 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:38.044446 (+ 300us) client_negotiation.cc:769] Sending connection context
0925 15:07:38.044701 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 15:07:38.045159 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":254,"threads_started":1}
I20260925 15:07:38.047699 11397 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:38.047991 11397 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:38.048625 11397 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
I20260925 15:07:38.049993 11074 ts_manager.cc:194] Registered new tserver with Master: f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:38.051261 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:60239
WARNING: no policy specified for oryx/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
I20260925 15:07:38.076404 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:38.379703 11402 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:38.379974 11402 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:38.380132 11402 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:38.386901 11402 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:38.387020 11402 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:38.387214 11402 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:38.398042 11402 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:37321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:38.399386 11402 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:38.400770 11402 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:38.412732 11407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:38.412813 11410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:38.412789 11408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:38.413518 11402 server_base.cc:1034] running on GCE node
I20260925 15:07:38.414155 11402 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:38.414964 11402 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:38.416177 11402 hybrid_clock.cc:648] HybridClock initialized: now 1790348858416137 us; error 49 us; skew 500 ppm
I20260925 15:07:38.416474 11402 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:38 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:38.419364 11402 init.cc:377] Logged in from keytab as oryx/127.4.205.195@KRBTEST.COM (short username oryx)
I20260925 15:07:38.420663 11402 webserver.cc:460] Webserver started at http://127.4.205.195:42403/ using document root <none> and password file <none>
I20260925 15:07:38.421308 11402 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:38.421448 11402 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:38.421787 11402 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:38.424453 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "dcbbd767bc14e0db3bdfe29c9cc18ccf"
server_key_iv: "a97092a40873293dfb853fedfed6f31c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.425173 11402 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "dcbbd767bc14e0db3bdfe29c9cc18ccf"
server_key_iv: "a97092a40873293dfb853fedfed6f31c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.429708 11402 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 15:07:38.432762 11417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:38.433899 11402 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:38.434052 11402 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "dcbbd767bc14e0db3bdfe29c9cc18ccf"
server_key_iv: "a97092a40873293dfb853fedfed6f31c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.434232 11402 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:38.449347 11402 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:38.452695 11402 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:38.452951 11402 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:38.453891 11402 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:38.455073 11402 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:38.455169 11402 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:38.455315 11402 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:38.455384 11402 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:38.486668 11402 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:32937
I20260925 15:07:38.486738 11529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:32937 every 8 connection(s)
I20260925 15:07:38.487788 11402 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:38.492085 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11402
I20260925 15:07:38.492250 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:38.492756 4919 external_mini_cluster.cc:1428] Setting key f691fd4d963ecaf111f5c8b6b6eba6e5
Sep 25 15:07:38 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:38.505798 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.490138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:59661 (local address 127.4.205.254:37321)
0925 15:07:38.490289 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:38.490293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:38.491271 (+ 978us) server_negotiation.cc:407] Connection header received
0925 15:07:38.493468 (+ 2197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:38.493475 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:38.493602 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:38.493714 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:38.495241 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.496086 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.497489 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.497959 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.498089 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:38.501159 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:38.501163 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:38.501166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:38.501191 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:38.503527 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:38.504117 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:38.504120 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:38.504122 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:38.504228 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:38.504552 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:38.504555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:38.504557 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:38.504771 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:38.504863 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:38.505377 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 15:07:38.505515 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:07:38.506908 11532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.490532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:59661)
0925 15:07:38.491116 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:38.491150 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:38.493201 (+ 2051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:38.493927 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:38.493935 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:38.494547 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:38.495056 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:38.495071 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.496228 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.496236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:38.497350 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:38.497361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.498161 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.498170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:38.498412 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:38.499044 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:38.499063 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:38.500913 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:38.503637 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:38.503643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:38.503652 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:38.504027 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:38.504317 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:38.504320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:38.504322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:38.504471 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:38.504875 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:38.504879 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:38.505124 (+ 245us) client_negotiation.cc:769] Sending connection context
0925 15:07:38.505314 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 15:07:38.505667 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":229,"threads_started":1}
I20260925 15:07:38.508044 11530 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:38.508316 11530 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:38.508965 11530 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
I20260925 15:07:38.510301 11074 ts_manager.cc:194] Registered new tserver with Master: dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:38.511008 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:59661
I20260925 15:07:38.519511 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.205.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.253.keytab.
Entry for principal oryx/127.4.205.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.253.keytab.
I20260925 15:07:38.563246 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:32803
--webserver_interface=127.4.205.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.253.keytab
--principal=oryx/127.4.205.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:37321,127.4.205.253:32803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:38.570714 11263 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
W20260925 15:07:38.863773 11535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:38.864081 11535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:38.864212 11535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:38.871136 11535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:38.871224 11535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:38.871268 11535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:38.871321 11535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:38.871364 11535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:38.880241 11535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:37321,127.4.205.253:32803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.253.keytab
--principal=oryx/127.4.205.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:32803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.205.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
W20260925 15:07:38.880501 11535 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.4.205.254:37321,127.4.205.253:32803'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 15:07:38.881390 11535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:38.882546 11535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:38.893476 11540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:38.893519 11543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:38.893582 11541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:38.894615 11535 server_base.cc:1034] running on GCE node
I20260925 15:07:38.895357 11535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:38.896559 11535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:38.897784 11535 hybrid_clock.cc:648] HybridClock initialized: now 1790348858897718 us; error 45 us; skew 500 ppm
I20260925 15:07:38.898130 11535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:38 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:38.901011 11535 init.cc:377] Logged in from keytab as oryx/127.4.205.253@KRBTEST.COM (short username oryx)
I20260925 15:07:38.902355 11535 webserver.cc:460] Webserver started at http://127.4.205.253:39471/ using document root <none> and password file <none>
I20260925 15:07:38.902981 11535 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:38.903103 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:38.903394 11535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:38.906136 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/instance:
uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "a6719fae9c716f46f92109b33616c3ae"
server_key_iv: "9e4b1e4353ec0b211015e604fc7d8c7c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.906857 11535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal/instance:
uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "a6719fae9c716f46f92109b33616c3ae"
server_key_iv: "9e4b1e4353ec0b211015e604fc7d8c7c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.911419 11535 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:38.914440 11550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:38.915606 11535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:38.915768 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "a6719fae9c716f46f92109b33616c3ae"
server_key_iv: "9e4b1e4353ec0b211015e604fc7d8c7c"
server_key_version: "encryptionkey@0"
I20260925 15:07:38.915962 11535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:38.966179 11535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:38.970320 11535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:38.970568 11535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:38.987150 11535 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.253:32803
I20260925 15:07:38.987169 11601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.253:32803 every 8 connection(s)
I20260925 15:07:38.988435 11535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
I20260925 15:07:38.990036 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11535
I20260925 15:07:38.992411 11602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:39.001271 11602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:38 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:39.011439 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:38.991950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36538 (local address 127.4.205.253:32803)
0925 15:07:38.992711 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:38.992722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:38.992759 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:07:38.994397 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:38.994417 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:38.994726 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:38.995078 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:38.996463 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.997793 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.999013 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:38.999624 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:38.999804 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.004297 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.004304 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.004317 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.004350 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.007625 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.008384 (+ 759us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.008393 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.008403 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.008576 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.008974 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.008981 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.008986 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.009412 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.009767 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.010191 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.010552 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":329,"threads_started":1}
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.253@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:39.017139 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.003830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45306)
0925 15:07:39.004441 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.004481 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.004619 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.005161 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.005165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.005893 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.006214 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.006222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.007471 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.007474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.008726 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.008737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.009457 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.009462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.009683 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.010242 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.010258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.012352 (+ 2094us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.015213 (+ 2861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.015221 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.015225 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.015590 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.015883 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.015886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.015888 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.016003 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.016432 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.016437 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.016552 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.016702 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.016888 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":267,"threads_started":1}
I20260925 15:07:39.017162 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.003887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.4.205.254:37321)
0925 15:07:39.004040 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.004044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.004582 (+ 538us) server_negotiation.cc:407] Connection header received
0925 15:07:39.004735 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.004740 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.004822 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.004937 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.006341 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.007349 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.008878 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.009263 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.009416 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.012550 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.012556 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.012560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.012596 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.015106 (+ 2510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.015679 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.015683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.015685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.015798 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.016085 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.016088 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.016090 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.016339 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.016443 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.016817 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.016966 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:07:39.018688 11602 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348858, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.253@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:39.030526 11606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.019385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36546)
0925 15:07:39.019527 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.019543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.019656 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.019902 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.019905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.020206 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.020306 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.020313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.021206 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.021212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.022233 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.022243 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.022909 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.022914 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.023093 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.023554 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.023568 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.025260 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.028576 (+ 3316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.028579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.028581 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.028903 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.029299 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.029302 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.029304 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.029432 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.029950 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.029953 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.030064 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.030162 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.030334 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:07:39.030812 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.019339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36546 (local address 127.4.205.253:32803)
0925 15:07:39.019521 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.019526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.019605 (+ 79us) server_negotiation.cc:407] Connection header received
0925 15:07:39.019745 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.019750 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.019806 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.019890 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.020418 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.021079 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.022371 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.022778 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.022899 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.025429 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.025435 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.025439 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.025491 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.028449 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.029009 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.029015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.029019 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.029192 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.029525 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.029530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.029533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.029849 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.029992 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.030268 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.030487 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 15:07:39.035463 11602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Bootstrap starting.
I20260925 15:07:39.038527 11602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:39.039451 11602 log.cc:824] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:39.041767 11602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: No bootstrap required, opened a new log
I20260925 15:07:39.048365 11602 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } }
I20260925 15:07:39.048630 11602 raft_consensus.cc:374] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:39.048700 11602 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f02e13a1c88741e3b3d0923b85e2f9cf, State: Initialized, Role: FOLLOWER
I20260925 15:07:39.049193 11602 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } }
I20260925 15:07:39.049945 11608 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } } }
I20260925 15:07:39.050271 11608 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:39.050853 11602 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:39.053769 11397 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
W20260925 15:07:39.059180 11619 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:07:39.059303 11619 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:07:39.061368 11535 master_runner.cc:428] Detected that this master f02e13a1c88741e3b3d0923b85e2f9cf is joining an existing cluster
I20260925 15:07:39.061476 11535 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.205.253:32803
I20260925 15:07:39.090512 11625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.065392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36552)
0925 15:07:39.074716 (+ 9324us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.074734 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.074879 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.075354 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.075361 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.076119 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.076464 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.076472 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.078135 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.078143 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.079932 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.079948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.081255 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.081267 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.081496 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.082155 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.082182 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.083578 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.087304 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.087310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.087315 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.088005 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.088390 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.088396 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.088400 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.088645 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.089378 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.089385 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.089526 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.089750 (+ 224us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.090059 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9175,"mutex_wait_us":45,"thread_start_us":7708,"threads_started":1}
I20260925 15:07:39.091362 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.069689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36552 (local address 127.4.205.253:32803)
0925 15:07:39.069866 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.069872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.074826 (+ 4954us) server_negotiation.cc:407] Connection header received
0925 15:07:39.075019 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.075027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.075125 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.075246 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.076594 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.077950 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.080059 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.080847 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.080962 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.083744 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.083756 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.083763 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.083800 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.087159 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.088145 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.088150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.088153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.088281 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.088771 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.088778 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.088787 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.089168 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.089328 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.090798 (+ 1470us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.091026 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":52}
I20260925 15:07:39.091847 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.065243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.4.205.254:37321)
0925 15:07:39.065902 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.065911 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.075498 (+ 9587us) server_negotiation.cc:407] Connection header received
0925 15:07:39.075627 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.075644 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.075724 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.075823 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.077146 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.078277 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.080290 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.080913 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.081008 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.083839 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.083845 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.083849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.083884 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.087436 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.089408 (+ 1972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.089413 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.089416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.089599 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.090066 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.090070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.090073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.090404 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.090540 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.091251 (+ 711us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.091480 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506}
I20260925 15:07:39.092828 11626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.065623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45320)
0925 15:07:39.075390 (+ 9767us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.075407 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.075521 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.075877 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.075882 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.076618 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.076976 (+ 358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.076987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.078478 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.078488 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.080118 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.080134 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.081310 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.081317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.081503 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.082345 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.082372 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.083611 (+ 1239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.088803 (+ 5192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.088807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.088810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.089285 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.089775 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.089779 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.089782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.089931 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.090761 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.090766 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.090960 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.092222 (+ 1262us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.092456 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9651,"mutex_wait_us":71,"spinlock_wait_cycles":20352,"thread_start_us":9471,"threads_started":1}
WARNING: no policy specified for oryx/127.4.205.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.252@KRBTEST.COM" created.
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.254@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:39.114804 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.098965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.4.205.253:32803)
0925 15:07:39.099122 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.099128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.102178 (+ 3050us) server_negotiation.cc:407] Connection header received
0925 15:07:39.102545 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.102552 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.102658 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.102780 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.103995 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.104703 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.105991 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.106385 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.106516 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.109496 (+ 2980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.109500 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.109504 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.109529 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.112172 (+ 2643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.112867 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.112874 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.112879 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.113026 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.113374 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.113381 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.113385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.113698 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.113840 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.114242 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.114439 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:07:39.115095 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.098933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36564)
0925 15:07:39.102052 (+ 3119us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.102099 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.102297 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.102856 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.102864 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.103612 (+ 748us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.103858 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.103870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.104831 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.104838 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.105855 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.105866 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.106558 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.106564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.106740 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.107293 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.107313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.109310 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.112354 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.112366 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.112375 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.112751 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.113135 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.113143 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.113148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.113274 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.113815 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.113823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.113992 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.114163 (+ 171us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.114459 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2905,"thread_start_us":2760,"threads_started":1}
I20260925 15:07:39.116767 11074 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.205.253:32803: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
member_type: NON_VOTER
last_known_addr {
host: "127.4.205.253"
port: 32803
}
attrs {
promote: true
}
}
dest_uuid: "fb955e793d8d4980b96753d75ce7903f"
cas_config_opid_index: -1
I20260925 15:07:39.117765 11074 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } }
I20260925 15:07:39.120721 11631 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } }
I20260925 15:07:39.121912 11577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 0 FOLLOWER]: Advancing to term 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.252.keytab.
Entry for principal oryx/127.4.205.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.252.keytab.
I20260925 15:07:39.125551 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:33555
--webserver_interface=127.4.205.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.252.keytab
--principal=oryx/127.4.205.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:39.126926 11577 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 15:07:39.131716 11632 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:39.132622 11631 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.205.253:32803
I20260925 15:07:39.133268 11535 master.cc:537] Master@127.4.205.253:32803 shutting down...
W20260925 15:07:39.135890 11060 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260925 15:07:39.138691 11634 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } }
I20260925 15:07:39.145836 11634 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
W20260925 15:07:39.148047 11060 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f -> Peer f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253:32803): Couldn't send request to peer f02e13a1c88741e3b3d0923b85e2f9cf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 15:07:39.151695 11535 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:39.152240 11535 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:39.152549 11535 tablet_replica.cc:330] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: stopping tablet replica
I20260925 15:07:39.160153 11535 master.cc:552] Master@127.4.205.253:32803 shutdown complete.
I20260925 15:07:39.160277 11535 master_runner.cc:305] Clearing existing system tablet
I20260925 15:07:39.160729 11535 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:39.160859 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:39.163962 11535 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:07:39.165719 11639 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:39.165987 11535 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:39.166204 11535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:07:39.166273 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "a6719fae9c716f46f92109b33616c3ae"
server_key_iv: "9e4b1e4353ec0b211015e604fc7d8c7c"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.170971 11535 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:39.177528 11535 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:07:39.177717 11535 log.cc:1196] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 15:07:39.178172 11535 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Deleting consensus metadata
I20260925 15:07:39.178426 11535 master_runner.cc:315] Copying system tablet from 127.4.205.254:37321
I20260925 15:07:39.180678 11535 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: tablet copy: Beginning tablet copy session from remote peer at address 127.4.205.254:37321
I20260925 15:07:39.196396 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.181419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45328 (local address 127.4.205.254:37321)
0925 15:07:39.181826 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.181832 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.182018 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:07:39.182274 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.182282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.182379 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.182512 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.183482 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.184626 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.186486 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.187107 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.187239 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.189592 (+ 2353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.189601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.189608 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.189655 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.193645 (+ 3990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.194379 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.194384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.194387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.194542 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.194959 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.194966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.194969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.195312 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.195476 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.195883 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.196096 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270}
I20260925 15:07:39.196499 11645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.181341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45328)
0925 15:07:39.181917 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.181937 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.182123 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.182553 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.182558 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.183071 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.183321 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.183333 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.184783 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.184789 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.186332 (+ 1543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.186346 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.187260 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.187266 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.187480 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.188166 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.188189 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.189278 (+ 1089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.193794 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.193800 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.193804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.194255 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.194667 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.194672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.194675 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.194821 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.195433 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.195438 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.195606 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.195748 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.196038 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":475,"threads_started":1}
I20260925 15:07:39.197290 11094 tablet_copy_service.cc:140] P fb955e793d8d4980b96753d75ce7903f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f02e13a1c88741e3b3d0923b85e2f9cf ({username='oryx', principal='oryx/127.4.205.253@KRBTEST.COM'} at 127.0.0.1:45328)
I20260925 15:07:39.197614 11094 tablet_copy_service.cc:161] P fb955e793d8d4980b96753d75ce7903f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f02e13a1c88741e3b3d0923b85e2f9cf at {username='oryx', principal='oryx/127.4.205.253@KRBTEST.COM'} at 127.0.0.1:45328: session id = f02e13a1c88741e3b3d0923b85e2f9cf-00000000000000000000000000000000
I20260925 15:07:39.201956 11094 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:07:39.203995 11535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:07:39.210530 11535 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: tablet copy: Starting download of 0 data blocks...
I20260925 15:07:39.210811 11535 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: tablet copy: Starting download of 1 WAL segments...
I20260925 15:07:39.213493 11535 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:07:39.217742 11094 tablet_copy_service.cc:342] P fb955e793d8d4980b96753d75ce7903f: Request end of tablet copy session f02e13a1c88741e3b3d0923b85e2f9cf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.205.253@KRBTEST.COM'} at 127.0.0.1:45328
I20260925 15:07:39.217919 11094 tablet_copy_service.cc:433] P fb955e793d8d4980b96753d75ce7903f: ending tablet copy session f02e13a1c88741e3b3d0923b85e2f9cf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f02e13a1c88741e3b3d0923b85e2f9cf
I20260925 15:07:39.229410 11535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:39.232106 11646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:39.232497 11647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:39.246997 11650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:39.247064 11535 server_base.cc:1034] running on GCE node
I20260925 15:07:39.247546 11535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:39.248270 11535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:39.249612 11535 hybrid_clock.cc:648] HybridClock initialized: now 1790348859249511 us; error 37 us; skew 500 ppm
I20260925 15:07:39.249910 11535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:39.252136 11535 init.cc:377] Logged in from keytab as oryx/127.4.205.253@KRBTEST.COM (short username oryx)
I20260925 15:07:39.253528 11535 webserver.cc:460] Webserver started at http://127.4.205.253:46015/ using document root <none> and password file <none>
I20260925 15:07:39.253973 11535 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:39.254124 11535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:39.257515 11535 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:07:39.259411 11656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:39.259850 11535 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:07:39.260035 11535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "a6719fae9c716f46f92109b33616c3ae"
server_key_iv: "9e4b1e4353ec0b211015e604fc7d8c7c"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.260260 11535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:39.270344 11535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:39.274362 11535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:39.295674 11535 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.253:32803
I20260925 15:07:39.295681 11707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.253:32803 every 8 connection(s)
I20260925 15:07:39.296823 11535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
I20260925 15:07:39.302071 11708 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:39.305879 11708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Bootstrap starting.
I20260925 15:07:39.320808 11708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:39.321272 11708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: Bootstrap complete.
I20260925 15:07:39.322278 11708 raft_consensus.cc:348] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } }
I20260925 15:07:39.322556 11708 raft_consensus.cc:729] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f02e13a1c88741e3b3d0923b85e2f9cf, State: Initialized, Role: LEARNER
I20260925 15:07:39.322772 11708 consensus_queue.cc:260] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } }
I20260925 15:07:39.323586 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } }
I20260925 15:07:39.323971 11708 sys_catalog.cc:564] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:39.324138 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: LEARNER
I20260925 15:07:39.332582 11722 catalog_manager.cc:1259] Loaded cluster ID: d168e20da74b422ba6f4e43e61800523
I20260925 15:07:39.332726 11722 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: loading cluster ID for follower catalog manager: success
I20260925 15:07:39.336283 11722 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: acquiring CA information for follower catalog manager: success
I20260925 15:07:39.339072 11722 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:07:39.508428 11633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:39.508661 11633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:39.508741 11633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 15:07:39.513123 11530 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
W20260925 15:07:39.519186 11633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:39.519297 11633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:39.519379 11633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:39.519457 11633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:39.519500 11633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:39.529495 11633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.252.keytab
--principal=oryx/127.4.205.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:33555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.205.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:39.531227 11633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:39.532492 11633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:39.543313 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:39.543491 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:39.543776 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:39.544937 11633 server_base.cc:1034] running on GCE node
I20260925 15:07:39.545536 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:39.546896 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:39.548086 11633 hybrid_clock.cc:648] HybridClock initialized: now 1790348859548048 us; error 33 us; skew 500 ppm
I20260925 15:07:39.548393 11633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:39.551692 11633 init.cc:377] Logged in from keytab as oryx/127.4.205.252@KRBTEST.COM (short username oryx)
I20260925 15:07:39.552999 11633 webserver.cc:460] Webserver started at http://127.4.205.252:37799/ using document root <none> and password file <none>
I20260925 15:07:39.553632 11633 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:39.553728 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:39.553962 11633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:39.556481 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/instance:
uuid: "1e46d6be33c04079919a8a6cb048baa4"
format_stamp: "Formatted at 2026-09-25 15:07:39 on dist-test-slave-64fx"
server_key: "06d9314c261bb6981f8ae206ebdb48de"
server_key_iv: "11b4ea0f6c9189781f2af9b00bcd36a1"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.557124 11633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal/instance:
uuid: "1e46d6be33c04079919a8a6cb048baa4"
format_stamp: "Formatted at 2026-09-25 15:07:39 on dist-test-slave-64fx"
server_key: "06d9314c261bb6981f8ae206ebdb48de"
server_key_iv: "11b4ea0f6c9189781f2af9b00bcd36a1"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.561888 11633 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260925 15:07:39.564961 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:39.566756 11633 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:39.566923 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "1e46d6be33c04079919a8a6cb048baa4"
format_stamp: "Formatted at 2026-09-25 15:07:39 on dist-test-slave-64fx"
server_key: "06d9314c261bb6981f8ae206ebdb48de"
server_key_iv: "11b4ea0f6c9189781f2af9b00bcd36a1"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.567114 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:39.571501 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.565341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36570)
0925 15:07:39.565523 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.565541 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.565672 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.566378 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.566382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.566405 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:39.566537 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.566544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.568310 (+ 1766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.568314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.569476 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.569486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.570994 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.570998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.571126 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.571162 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.571236 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.571321 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:07:39.571853 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.565526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36570 (local address 127.4.205.253:32803)
0925 15:07:39.566055 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.566062 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.566079 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:39.566145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.566150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.566225 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.566351 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:39.566662 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.568174 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.569624 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.570844 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.570966 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.571181 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.571257 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.571391 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":239,"threads_started":1}
I20260925 15:07:39.572991 11683 raft_consensus.cc:1203] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260925 15:07:39.573719 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb955e793d8d4980b96753d75ce7903f. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } }
I20260925 15:07:39.573951 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: LEARNER
I20260925 15:07:39.582772 11633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:39.586057 11633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:39.586254 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:39.604681 11633 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.252:33555
I20260925 15:07:39.604698 11789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.252:33555 every 8 connection(s)
I20260925 15:07:39.605923 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
I20260925 15:07:39.609037 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11633
I20260925 15:07:39.609418 11790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:39.618831 11790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.252@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:39.633116 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.611110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50770 (local address 127.4.205.252:33555)
0925 15:07:39.611879 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.611898 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.611945 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:07:39.613909 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.613933 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.614405 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.615178 (+ 773us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.616494 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.618348 (+ 1854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.619544 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.620427 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.620589 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.625307 (+ 4718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.625322 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.625331 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.625360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.629316 (+ 3956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.630101 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.630107 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.630112 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.630245 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.630612 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.630622 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.630626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.631168 (+ 542us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.631449 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.631867 (+ 418us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.632250 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":317,"threads_started":1}
I20260925 15:07:39.636598 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.621340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45330)
0925 15:07:39.622045 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.622087 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.622269 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.622676 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.622680 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.623196 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.623401 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.623408 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.624719 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.624724 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.626100 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.626132 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.627492 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.627499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.627696 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.628229 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.628250 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.630922 (+ 2672us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.634035 (+ 3113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.634046 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.634053 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.634471 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.634809 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.634819 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.634823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.634952 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.635592 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.635598 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.635771 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.635943 (+ 172us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.636217 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":371,"threads_started":1}
I20260925 15:07:39.636636 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.621600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.4.205.254:37321)
0925 15:07:39.621789 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.621794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.622164 (+ 370us) server_negotiation.cc:407] Connection header received
0925 15:07:39.622424 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.622444 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.622528 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.622637 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.623554 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.624538 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.626272 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.627294 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.627410 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.631095 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.631099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.631102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.631128 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.633920 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.634592 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.634603 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.634605 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.634708 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.635093 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.635098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.635100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.635463 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.635604 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.636045 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.636249 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:07:39.637823 11790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.252@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:39.649120 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.638503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36576 (local address 127.4.205.253:32803)
0925 15:07:39.638670 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.638677 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.638703 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:07:39.638750 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.638753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.638843 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.638936 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.639467 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.640118 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.641176 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.641611 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.641710 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.644186 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.644191 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.644193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.644220 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.647193 (+ 2973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.647727 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.647730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.647732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.647849 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.648145 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.648148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.648150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.648376 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.648475 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.648718 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.648893 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:07:39.649202 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.638380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36576)
0925 15:07:39.638576 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.638595 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.638681 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.638972 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.638975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.639244 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.639356 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.639362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.640215 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.640218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.641067 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.641077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.641743 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.641747 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.641855 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.642292 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.642307 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.644030 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.647294 (+ 3264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.647298 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.647300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.647646 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.647943 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.647946 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.647948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.648061 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.648466 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.648469 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.648567 (+ 98us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.648669 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.648841 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 15:07:39.650192 11790 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.252@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
I20260925 15:07:39.660645 11794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.650801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:50776)
0925 15:07:39.650984 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.651000 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.651104 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.651359 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.651362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.651651 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.651771 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.651778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.652586 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.652589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.653330 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.653339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.654020 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.654024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.654148 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.654584 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.654598 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.656241 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.658763 (+ 2522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.658766 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.658768 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.659098 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.659376 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.659379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.659381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.659504 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.659899 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.659902 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.660014 (+ 112us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.660112 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.660316 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:07:39.660645 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.651000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50776 (local address 127.4.205.252:33555)
0925 15:07:39.651143 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.651147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.651186 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:07:39.651218 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.651221 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.651265 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.651332 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.651878 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.652493 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.653433 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.653864 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.653977 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.656383 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.656387 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.656390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.656414 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.658665 (+ 2251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.659186 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.659189 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.659191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.659283 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.659586 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.659589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.659590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.659804 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.659907 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.660235 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.660425 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":66}
I20260925 15:07:39.665853 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Bootstrap starting.
I20260925 15:07:39.668682 11790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:39.669698 11790 log.cc:824] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:39.672097 11790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: No bootstrap required, opened a new log
I20260925 15:07:39.678795 11790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } }
I20260925 15:07:39.679075 11790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:39.679149 11790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e46d6be33c04079919a8a6cb048baa4, State: Initialized, Role: FOLLOWER
I20260925 15:07:39.679661 11790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } }
I20260925 15:07:39.680444 11796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } } }
I20260925 15:07:39.680760 11796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:39.681360 11790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:39.687281 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11135
W20260925 15:07:39.690614 11807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:07:39.690785 11807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:07:39.694248 11633 master_runner.cc:428] Detected that this master 1e46d6be33c04079919a8a6cb048baa4 is joining an existing cluster
I20260925 15:07:39.694367 11633 master_runner.cc:432] Initiating AddMaster RPC to add 127.4.205.252:33555
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.193@KRBTEST.COM
I20260925 15:07:39.709378 11809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.688908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56220 (local address 127.4.205.193:39305)
0925 15:07:39.689846 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.689855 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.689899 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:07:39.689960 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.689969 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.690192 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.690994 (+ 802us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.691886 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.693377 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.695304 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.696110 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.701885 (+ 5775us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.702119 (+ 234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.702129 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.702144 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.702176 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.705648 (+ 3472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.706376 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.706383 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.706387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.706515 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.706942 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.706949 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.706953 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.707314 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.708577 (+ 1263us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.708727 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.708969 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":1683,"threads_started":1}
I20260925 15:07:39.710561 11258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:39.716573 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.699162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.4.205.254:37321)
0925 15:07:39.699342 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.699350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.699369 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:07:39.699416 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.699433 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.699507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.699605 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.701009 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.702369 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.704757 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.705545 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.705685 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.708883 (+ 3198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.708892 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.708900 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.708943 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.713026 (+ 4083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.713976 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.713984 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.713990 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.714160 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.714688 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.714695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.714701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.715083 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.715230 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.715883 (+ 653us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.716127 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:07:39.717738 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.698139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45334)
0925 15:07:39.698745 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.698763 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.698876 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.699769 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.699773 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.700433 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.700843 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.700856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.702592 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.702600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.704576 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.704592 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.706371 (+ 1779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.706379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.706616 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.707239 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.707266 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.708639 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.713221 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.713229 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.713235 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.713825 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.714333 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.714340 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.714346 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.714547 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.715334 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.715340 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.715587 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.716778 (+ 1191us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.717119 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":2815,"threads_started":1}
I20260925 15:07:39.736029 11738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.699821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.4.205.253:32803)
0925 15:07:39.700168 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.700174 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.718479 (+ 18305us) server_negotiation.cc:407] Connection header received
0925 15:07:39.718718 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.718726 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.718819 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.718918 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.720630 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.721984 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.725709 (+ 3725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.726386 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.726472 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.728871 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.728877 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.728882 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.728920 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.732811 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.733764 (+ 953us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.733769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.733773 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.733903 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.734361 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.734365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.734368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.734669 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.734827 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.735448 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.735702 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231}
I20260925 15:07:39.737468 11814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.698745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:36580)
0925 15:07:39.718384 (+ 19639us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.718405 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.718575 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.719568 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.719575 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.720177 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.720441 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.720455 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.722203 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.722210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.725504 (+ 3294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.725514 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.726757 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.726764 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.726941 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.727503 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.727522 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.728683 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.732965 (+ 4282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.732970 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.732973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.733538 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.734047 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.734053 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.734059 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.734217 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.734952 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.734959 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.735105 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.736311 (+ 1206us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.737110 (+ 799us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19497,"thread_start_us":31100,"threads_started":1}
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.254@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
I20260925 15:07:39.740355 11793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.699993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50786 (local address 127.4.205.252:33555)
0925 15:07:39.700300 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.700305 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.724926 (+ 24621us) server_negotiation.cc:407] Connection header received
0925 15:07:39.725485 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.725489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.725604 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.725668 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.726828 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.727960 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.730108 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.730891 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.730983 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.733429 (+ 2446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.733434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.733438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.733489 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.736773 (+ 3284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.737445 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.737450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.737454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.737870 (+ 416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.738879 (+ 1009us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.738885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.738894 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.739210 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.739290 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.739626 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.739951 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":198}
I20260925 15:07:39.740391 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.699437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:50786)
0925 15:07:39.724845 (+ 25408us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.724864 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.725289 (+ 425us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.725841 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.725845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.726451 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.726702 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.726713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.728209 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.728214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.729776 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.729789 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.731242 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.731249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.731418 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.732304 (+ 886us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.732324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.733253 (+ 929us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.736878 (+ 3625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.736885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.736892 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.737335 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.738115 (+ 780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.738119 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.738122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.738746 (+ 624us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.739340 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.739342 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.739438 (+ 96us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.739564 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.739836 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25200,"thread_start_us":24874,"threads_started":1}
I20260925 15:07:39.747881 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.723719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:50788)
0925 15:07:39.723908 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.723931 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.724043 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.729157 (+ 5114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.729162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.729545 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.729760 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.729771 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.733458 (+ 3687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.733464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.734725 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.734748 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.736674 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.736680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.736868 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.737470 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.737494 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.740939 (+ 3445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.745675 (+ 4736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.745692 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.745696 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.746055 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.746438 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.746443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.746447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.746565 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.747061 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.747065 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.747194 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.747313 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.747537 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:07:39.747925 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.724045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50788 (local address 127.4.205.252:33555)
0925 15:07:39.728011 (+ 3966us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.728016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.728039 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:39.728114 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.728119 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.728221 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.729916 (+ 1695us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.730227 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.731371 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.734981 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.736237 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.741141 (+ 4904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.741847 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.741856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.741862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.741898 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.745492 (+ 3594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.746172 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.746178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.746183 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.746322 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.746671 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.746676 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.746680 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.746943 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.747085 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.747399 (+ 314us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.747548 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3828,"thread_start_us":3682,"threads_started":1}
I20260925 15:07:39.749035 11074 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.4.205.252:33555: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4"
member_type: NON_VOTER
last_known_addr {
host: "127.4.205.252"
port: 33555
}
attrs {
promote: true
}
}
dest_uuid: "fb955e793d8d4980b96753d75ce7903f"
cas_config_opid_index: 5
I20260925 15:07:39.749922 11074 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } }
I20260925 15:07:39.751574 11634 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.752107 11683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:07:39.752923 11765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:39.756115 11634 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.4.205.252:33555
I20260925 15:07:39.756474 11765 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 15:07:39.756783 11633 master.cc:537] Master@127.4.205.252:33555 shutting down...
I20260925 15:07:39.764794 11820 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 15:07:39.767464 11819 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:39.769804 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.770125 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:39.778079 11683 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.781268 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: NON_VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: true } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.781687 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: LEARNER
I20260925 15:07:39.782163 11820 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: attempting to promote NON_VOTER f02e13a1c88741e3b3d0923b85e2f9cf to VOTER
I20260925 15:07:39.782940 11820 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } }
I20260925 15:07:39.785085 11683 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 LEARNER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 15:07:39.785677 11821 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 15:07:39.788165 11633 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:39.788667 11633 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:39.788908 11633 tablet_replica.cc:330] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: stopping tablet replica
I20260925 15:07:39.798353 11633 master.cc:552] Master@127.4.205.252:33555 shutdown complete.
I20260925 15:07:39.798518 11633 master_runner.cc:305] Clearing existing system tablet
I20260925 15:07:39.799113 11633 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:39.799300 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:39.804867 11821 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.810775 11682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.811074 11819 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.811347 11819 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:39.811695 11819 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.811908 11819 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:39.812736 11633 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.003s sys 0.000s
I20260925 15:07:39.815613 11711 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:39.815927 11711 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:39.818903 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.818329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:50796)
0925 15:07:39.818527 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.818670 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:33555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:07:39.819327 11059 consensus_peers.cc:487] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f -> Peer 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252:33555): Couldn't send request to peer 1e46d6be33c04079919a8a6cb048baa4. Status: Network error: Client connection negotiation failed: client connection to 127.4.205.252:33555: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:07:39.825896 11826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:39.826218 11633 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:39.826539 11633 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.003s sys 0.008s
I20260925 15:07:39.826651 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "1e46d6be33c04079919a8a6cb048baa4"
format_stamp: "Formatted at 2026-09-25 15:07:39 on dist-test-slave-64fx"
server_key: "06d9314c261bb6981f8ae206ebdb48de"
server_key_iv: "11b4ea0f6c9189781f2af9b00bcd36a1"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.830228 11633 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:39.834777 11633 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 15:07:39.834946 11633 log.cc:1196] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 15:07:39.835462 11633 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Deleting consensus metadata
I20260925 15:07:39.835703 11633 master_runner.cc:315] Copying system tablet from 127.4.205.254:37321
I20260925 15:07:39.842010 11633 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: tablet copy: Beginning tablet copy session from remote peer at address 127.4.205.254:37321
I20260925 15:07:39.859534 11265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.842765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.4.205.254:37321)
0925 15:07:39.842988 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:39.842994 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:39.843396 (+ 402us) server_negotiation.cc:407] Connection header received
0925 15:07:39.843614 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:39.843621 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:39.843702 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:39.843812 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:39.844586 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.845312 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.846627 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.847049 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.847180 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.853872 (+ 6692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:39.853876 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:39.853880 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:39.853909 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:39.856952 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.857629 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.857639 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.857641 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.857757 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:39.858159 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:39.858165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:39.858169 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:39.858432 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:39.858592 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:39.858967 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 15:07:39.859193 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 15:07:39.859846 11832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:39.842749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:45340)
0925 15:07:39.843293 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:39.843309 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:39.843468 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:39.843827 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:39.843831 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:39.844252 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:39.844441 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.844449 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.845493 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.845499 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.846446 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:39.846457 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:39.847196 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:39.847201 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:39.847358 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:39.852822 (+ 5464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:39.852840 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:39.853677 (+ 837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:39.857083 (+ 3406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.857090 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.857095 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.857460 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.857880 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:39.857885 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:39.857890 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:39.858050 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:39.858549 (+ 499us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:39.858555 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:39.858722 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:07:39.858873 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:07:39.859197 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":273,"threads_started":1}
I20260925 15:07:39.860394 11094 tablet_copy_service.cc:140] P fb955e793d8d4980b96753d75ce7903f: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1e46d6be33c04079919a8a6cb048baa4 ({username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:45340)
I20260925 15:07:39.860570 11094 tablet_copy_service.cc:161] P fb955e793d8d4980b96753d75ce7903f: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1e46d6be33c04079919a8a6cb048baa4 at {username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:45340: session id = 1e46d6be33c04079919a8a6cb048baa4-00000000000000000000000000000000
I20260925 15:07:39.863096 11094 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 15:07:39.864811 11633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 15:07:39.869129 11633 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: tablet copy: Starting download of 0 data blocks...
I20260925 15:07:39.869411 11633 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: tablet copy: Starting download of 1 WAL segments...
I20260925 15:07:39.872481 11633 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 15:07:39.875480 11094 tablet_copy_service.cc:342] P fb955e793d8d4980b96753d75ce7903f: Request end of tablet copy session 1e46d6be33c04079919a8a6cb048baa4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:45340
I20260925 15:07:39.875599 11094 tablet_copy_service.cc:433] P fb955e793d8d4980b96753d75ce7903f: ending tablet copy session 1e46d6be33c04079919a8a6cb048baa4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1e46d6be33c04079919a8a6cb048baa4
I20260925 15:07:39.885208 11633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:07:39.888239 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11135
W20260925 15:07:39.891250 11833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:39.894953 11834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:39.909729 11633 server_base.cc:1034] running on GCE node
W20260925 15:07:39.910640 11836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:39.911059 11633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:39.911486 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:39305
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:37321
--webserver_port=33265
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:39.912010 11633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:39.915318 11633 hybrid_clock.cc:648] HybridClock initialized: now 1790348859915272 us; error 40 us; skew 500 ppm
I20260925 15:07:39.915541 11633 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:39 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348859, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:39.917330 11633 init.cc:377] Logged in from keytab as oryx/127.4.205.252@KRBTEST.COM (short username oryx)
I20260925 15:07:39.918632 11633 webserver.cc:460] Webserver started at http://127.4.205.252:37197/ using document root <none> and password file <none>
I20260925 15:07:39.918921 11633 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:39.919035 11633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:39.921447 11633 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:07:39.923069 11844 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:39.923450 11633 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:07:39.923566 11633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "1e46d6be33c04079919a8a6cb048baa4"
format_stamp: "Formatted at 2026-09-25 15:07:39 on dist-test-slave-64fx"
server_key: "06d9314c261bb6981f8ae206ebdb48de"
server_key_iv: "11b4ea0f6c9189781f2af9b00bcd36a1"
server_key_version: "encryptionkey@0"
I20260925 15:07:39.923739 11633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:39.940142 11633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:39.943349 11633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:39.964036 11633 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.252:33555
I20260925 15:07:39.964113 11895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.252:33555 every 8 connection(s)
I20260925 15:07:39.964953 11633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
I20260925 15:07:39.970111 11896 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:39.974007 11896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Bootstrap starting.
I20260925 15:07:39.996116 11896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:39.996697 11896 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Bootstrap complete.
I20260925 15:07:39.997802 11896 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } }
I20260925 15:07:39.998060 11896 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1e46d6be33c04079919a8a6cb048baa4, State: Initialized, Role: LEARNER
I20260925 15:07:39.998317 11896 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } }
I20260925 15:07:40.010005 11899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:40.010334 11896 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:40.010332 11899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: LEARNER
I20260925 15:07:40.026901 11910 catalog_manager.cc:1259] Loaded cluster ID: d168e20da74b422ba6f4e43e61800523
I20260925 15:07:40.026984 11910 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: loading cluster ID for follower catalog manager: success
I20260925 15:07:40.031204 11910 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: acquiring CA information for follower catalog manager: success
I20260925 15:07:40.033316 11910 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:07:40.268296 11840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:40.268582 11840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:40.268744 11840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:40.276480 11840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:40.276613 11840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:40.276835 11840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:40.286055 11840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:39305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=33265
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:40.287340 11840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:40.288759 11840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:07:40.294680 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.289050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50800 (local address 127.4.205.252:33555)
0925 15:07:40.289536 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.289540 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.289601 (+ 61us) server_negotiation.cc:407] Connection header received
0925 15:07:40.289669 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.289673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.289741 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.289834 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:40.290217 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.291112 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.293069 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.293877 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.293981 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.294114 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:40.294369 (+ 255us) server_negotiation.cc:299] Negotiation successful
0925 15:07:40.294457 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":258,"threads_started":1}
I20260925 15:07:40.294728 11628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.288876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:50800)
0925 15:07:40.289107 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.289128 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:40.289290 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:40.289870 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:40.289875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:40.289903 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:40.290078 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.290089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.291244 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.291249 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.292923 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.292938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.294036 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.294041 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.294205 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.294238 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:07:40.294343 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:07:40.294467 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 15:07:40.295506 11871 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260925 15:07:40.296043 11899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb955e793d8d4980b96753d75ce7903f. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
I20260925 15:07:40.296341 11899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: LEARNER
W20260925 15:07:40.303767 11917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:40.304262 11916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:40.304965 11919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:40.305332 11840 server_base.cc:1034] running on GCE node
I20260925 15:07:40.305974 11840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:40.306814 11840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:40.308053 11840 hybrid_clock.cc:648] HybridClock initialized: now 1790348860308005 us; error 39 us; skew 500 ppm
I20260925 15:07:40.308367 11840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:40.311544 11840 init.cc:377] Logged in from keytab as oryx/127.4.205.193@KRBTEST.COM (short username oryx)
I20260925 15:07:40.313062 11840 webserver.cc:460] Webserver started at http://127.4.205.193:33265/ using document root <none> and password file <none>
I20260925 15:07:40.313751 11840 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:40.313907 11840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:40.318899 11840 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:07:40.321436 11926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.322747 11840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:40.322942 11840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "8987d797c4c14d8f80a376be1d245505"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "fcda735708615dad6dbd6bc8f1213c9c"
server_key_iv: "e361d6031787172cd58a5e362a8d1d9d"
server_key_version: "encryptionkey@0"
I20260925 15:07:40.323947 11840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:40.336862 11840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:40.341080 11840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:40.341310 11840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:40.342146 11840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:40.343315 11840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:40.343393 11840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.343461 11840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:40.343511 11840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.377254 11840 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:39305
I20260925 15:07:40.377342 12038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:39305 every 8 connection(s)
I20260925 15:07:40.378654 11840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:40.384349 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 11840
I20260925 15:07:40.384739 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11269
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.194@KRBTEST.COM
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:40.435587 11914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.381770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:47569 (local address 127.4.205.252:33555)
0925 15:07:40.381930 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.381934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.406724 (+ 24790us) server_negotiation.cc:407] Connection header received
0925 15:07:40.412044 (+ 5320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.412053 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.412185 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.412273 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:40.415211 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.416639 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.419962 (+ 3323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.420825 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.420941 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.428485 (+ 7544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:40.428495 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:40.428503 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:40.428556 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:40.431950 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.432908 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.432914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.432918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.433040 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.433538 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.433545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.433551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.434010 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:40.434130 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:40.434979 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 15:07:40.435182 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:07:40.436672 12044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.382391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.4.205.193:47569)
0925 15:07:40.406522 (+ 24131us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.406588 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:40.410091 (+ 3503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:40.412962 (+ 2871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:40.412977 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:40.414004 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:40.414979 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.415007 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.416894 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.416902 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.419771 (+ 2869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.419789 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.421431 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.421445 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.421873 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.423074 (+ 1201us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:40.423125 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:40.427655 (+ 4530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:40.432122 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.432128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.432139 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.432749 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.433182 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.433186 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.433190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.433417 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.434171 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:40.434179 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:40.434587 (+ 408us) client_negotiation.cc:769] Sending connection context
0925 15:07:40.434869 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 15:07:40.435333 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23717,"mutex_wait_us":1,"thread_start_us":30214,"threads_started":1}
I20260925 15:07:40.439625 12039 heartbeater.cc:347] Connected to a master server at 127.4.205.252:33555
I20260925 15:07:40.439951 12039 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:40.440686 12049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.385632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46532 (local address 127.4.205.194:33547)
0925 15:07:40.414428 (+ 28796us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.414439 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.414472 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:07:40.414542 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.414553 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.414821 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.418701 (+ 3880us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:40.418935 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.420378 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.423360 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.424200 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.428326 (+ 4126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.432137 (+ 3811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:40.432150 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:40.432166 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:40.432207 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:40.436270 (+ 4063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.437262 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.437272 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.437332 (+ 60us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.437522 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.437963 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.437970 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.437975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.438344 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:40.439885 (+ 1541us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:40.440046 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:07:40.440299 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28624,"thread_start_us":28481,"threads_started":1}
I20260925 15:07:40.440829 12043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.382391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.4.205.193:37407)
0925 15:07:40.412497 (+ 30106us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.412524 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:40.412655 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:40.420549 (+ 7894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:40.420554 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:40.421122 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:40.421460 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.421472 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.422917 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.422934 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.426598 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.426614 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.430645 (+ 4031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.430651 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.430822 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.431314 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:40.431334 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:40.433713 (+ 2379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:40.437574 (+ 3861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.437583 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.437588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.438508 (+ 920us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.438917 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.438925 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.438931 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.439107 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.439821 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:40.439826 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:40.439983 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:07:40.440110 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:07:40.440375 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29918,"mutex_wait_us":50,"spinlock_wait_cycles":19712,"thread_start_us":27866,"threads_started":1}
I20260925 15:07:40.440882 12039 heartbeater.cc:511] Master 127.4.205.252:33555 requested a full tablet report, sending...
I20260925 15:07:40.441704 12046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.383499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37407 (local address 127.4.205.253:32803)
0925 15:07:40.415302 (+ 31803us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.415311 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.415339 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:07:40.415408 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.415415 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.415513 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.415639 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:40.421664 (+ 6025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.422757 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.426888 (+ 4131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.427649 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.431071 (+ 3422us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.433924 (+ 2853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:40.433930 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:40.433934 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:40.433968 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:40.437412 (+ 3444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.438637 (+ 1225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.438642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.438645 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.438784 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.439261 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.439280 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.439283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.439617 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:40.439761 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:40.441121 (+ 1360us) server_negotiation.cc:299] Negotiation successful
0925 15:07:40.441296 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31658,"thread_start_us":31567,"threads_started":1}
I20260925 15:07:40.443137 11392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:40.443418 12045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.382923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:60785)
0925 15:07:40.416237 (+ 33314us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:40.416253 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:40.416346 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:40.417354 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:40.417358 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:40.418068 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:40.418439 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.418452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.420087 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.420093 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.424538 (+ 4445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:40.424552 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.425810 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.425815 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:40.426024 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.428363 (+ 2339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:40.428385 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:40.433192 (+ 4807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:40.440687 (+ 7495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.440692 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.440695 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.441262 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.441736 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:40.441741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:40.441747 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:40.441896 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:40.442480 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:40.442484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:40.442625 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 15:07:40.442751 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:07:40.443028 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33197,"mutex_wait_us":2788,"thread_start_us":33047,"threads_started":1}
I20260925 15:07:40.443444 12047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.383797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:60785 (local address 127.4.205.254:37321)
0925 15:07:40.413964 (+ 30167us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.413972 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.416450 (+ 2478us) server_negotiation.cc:407] Connection header received
0925 15:07:40.416520 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.416526 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.416604 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.416702 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:40.418647 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.419918 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.424707 (+ 4789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:40.425452 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:40.425594 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:40.433392 (+ 7798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:40.433398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:40.433403 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:40.433453 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:40.440526 (+ 7073us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.441401 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.441406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.441410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.441547 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:40.442023 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:40.442029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:40.442040 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:40.442361 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:40.442501 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:40.442864 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 15:07:40.443059 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30030,"thread_start_us":29957,"threads_started":1}
I20260925 15:07:40.443615 11861 ts_manager.cc:194] Registered new tserver with Master: 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305)
I20260925 15:07:40.444113 12041 heartbeater.cc:347] Connected to a master server at 127.4.205.253:32803
I20260925 15:07:40.444257 12041 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:40.444429 12040 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:40.444564 12040 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:40.444741 12041 heartbeater.cc:511] Master 127.4.205.253:32803 requested a full tablet report, sending...
I20260925 15:07:40.445052 12040 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
I20260925 15:07:40.446431 11074 ts_manager.cc:194] Re-registered known tserver with Master: 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305)
I20260925 15:07:40.446496 11673 ts_manager.cc:194] Registered new tserver with Master: 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305)
I20260925 15:07:40.447345 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:60785
I20260925 15:07:40.529063 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11269
I20260925 15:07:40.545219 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:33547
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:37321
--webserver_port=40033
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:40.850625 12051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:40.850865 12051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:40.850950 12051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 15:07:40.857604 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: NON_VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: true } } }
W20260925 15:07:40.857908 12051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
I20260925 15:07:40.857900 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
W20260925 15:07:40.858013 12051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:40.858295 12051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:40.858919 12055 raft_consensus.cc:1050] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: attempting to promote NON_VOTER 1e46d6be33c04079919a8a6cb048baa4 to VOTER
I20260925 15:07:40.859445 12055 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } }
I20260925 15:07:40.861447 11871 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 LEARNER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:07:40.861634 11682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Refusing update from remote peer fb955e793d8d4980b96753d75ce7903f: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 15:07:40.862108 12055 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:07:40.862493 12056 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 15:07:40.865880 11682 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.866572 11871 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.865296 12055 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.871269 12061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.871644 12061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:40.872809 12051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:33547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=40033
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:40.874018 12062 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.874431 12062 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:40.874289 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.874514 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:40.874485 12051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:40.874950 12055 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.875154 12055 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:40.875044 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fb955e793d8d4980b96753d75ce7903f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:40.875296 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:40.875960 12051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:40.888779 12064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:40.888821 12065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:40.888793 12067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:40.889544 12051 server_base.cc:1034] running on GCE node
I20260925 15:07:40.890136 12051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:40.890957 12051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:40.892143 12051 hybrid_clock.cc:648] HybridClock initialized: now 1790348860892075 us; error 59 us; skew 500 ppm
I20260925 15:07:40.892386 12051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:40 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:40.895262 12051 init.cc:377] Logged in from keytab as oryx/127.4.205.194@KRBTEST.COM (short username oryx)
I20260925 15:07:40.896710 12051 webserver.cc:460] Webserver started at http://127.4.205.194:40033/ using document root <none> and password file <none>
I20260925 15:07:40.897341 12051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:40.897485 12051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:40.902673 12051 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:40.905184 12074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.906471 12051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:07:40.906634 12051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
format_stamp: "Formatted at 2026-09-25 15:07:37 on dist-test-slave-64fx"
server_key: "0463e3581745fffe395cdbc901b111a8"
server_key_iv: "d641609e5365f5de9246ceddb4c8280a"
server_key_version: "encryptionkey@0"
I20260925 15:07:40.907557 12051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:40.923570 12051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:40.928120 12051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:40.928458 12051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:40.929349 12051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:40.930648 12051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:40.930765 12051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.930869 12051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:40.930938 12051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:40.962587 12051 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:33547
I20260925 15:07:40.962630 12186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:33547 every 8 connection(s)
I20260925 15:07:40.964049 12051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:40.973253 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12051
I20260925 15:07:40.973726 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11402
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348856, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.195@KRBTEST.COM
I20260925 15:07:41.021939 12198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.974860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.4.205.195:32937)
0925 15:07:40.990454 (+ 15594us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.990466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:40.991335 (+ 869us) server_negotiation.cc:407] Connection header received
0925 15:07:40.991444 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:40.991456 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:40.991772 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:40.991940 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:40.998082 (+ 6142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.000215 (+ 2133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.002809 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.003968 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.004092 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.012295 (+ 8203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.012315 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.012340 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.012394 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.017373 (+ 4979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.018516 (+ 1143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.018530 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.018536 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.018738 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.019274 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.019281 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.019287 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.019786 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.020145 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.020908 (+ 763us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.021304 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15374,"thread_start_us":15600,"threads_started":1}
I20260925 15:07:41.024210 11525 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348860, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
I20260925 15:07:41.067006 12190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.966808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:55395 (local address 127.4.205.252:33555)
0925 15:07:40.967307 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:40.967313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.032762 (+ 65449us) server_negotiation.cc:407] Connection header received
0925 15:07:41.041130 (+ 8368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.041139 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.041264 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.041360 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.048275 (+ 6915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.049907 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.052751 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.053419 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.053509 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.059807 (+ 6298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.059817 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.059824 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.059868 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.063408 (+ 3540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.064252 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.064257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.064261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.064427 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.065142 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.065147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.065150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.065552 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.065676 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.066446 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.066650 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":968,"threads_started":1}
I20260925 15:07:41.068213 12191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.967508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.4.205.194:55395)
0925 15:07:41.032640 (+ 65132us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.032664 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.040957 (+ 8293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.042026 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.042043 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.043190 (+ 1147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.044197 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.044226 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.050160 (+ 5934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.050168 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.052596 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.052608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.053871 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.053879 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.054603 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.055388 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.055434 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.059595 (+ 4161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.063569 (+ 3974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.063573 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.063576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.064140 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.064559 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.064563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.064567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.065019 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.065752 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.065755 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.066005 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.067475 (+ 1470us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.067711 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64972,"mutex_wait_us":21,"spinlock_wait_cycles":63232,"thread_start_us":82985,"threads_started":1}
I20260925 15:07:41.068444 12196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.972494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:50857 (local address 127.4.205.253:32803)
0925 15:07:41.030903 (+ 58409us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.030912 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.051036 (+ 20124us) server_negotiation.cc:407] Connection header received
0925 15:07:41.051193 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.051198 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.051284 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.051398 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.052674 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.053763 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.055999 (+ 2236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.056574 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.056674 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.061862 (+ 5188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.061868 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.061873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.061935 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.065702 (+ 3767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.066442 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.066447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.066451 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.066570 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.066976 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.066981 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.066984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.067279 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.067393 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.067894 (+ 501us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.068102 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58225,"thread_start_us":58411,"threads_started":1}
I20260925 15:07:41.068918 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.967864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:46303 (local address 127.4.205.254:37321)
0925 15:07:41.044780 (+ 76916us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.044789 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.044811 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:07:41.044893 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.044900 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.044990 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.045116 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.046552 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.048047 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.054925 (+ 6878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.055573 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.055683 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.059608 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.059615 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.059619 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.059649 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.063218 (+ 3569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.064252 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.064257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.064261 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.064437 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.064931 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.064938 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.064941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.065265 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.065406 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.068302 (+ 2896us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.068516 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76744,"thread_start_us":76620,"threads_started":1}
I20260925 15:07:41.069293 12192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.972002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.4.205.194:50857)
0925 15:07:41.050890 (+ 78888us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.050915 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.051072 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.051564 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.051569 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.052150 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.052463 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.052474 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.053910 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.053914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.055826 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.055839 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.056983 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.056990 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.057141 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.057742 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.057781 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.060333 (+ 2552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.065857 (+ 5524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.065862 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.065866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.066323 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.066690 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.066694 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.066698 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.066870 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.067479 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.067483 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.067609 (+ 126us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.068651 (+ 1042us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.068912 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78742,"thread_start_us":59023,"threads_started":1}
I20260925 15:07:41.069635 12195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:40.967475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:46303)
0925 15:07:41.027523 (+ 60048us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.027592 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.040521 (+ 12929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.045351 (+ 4830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.045358 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.046006 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.046347 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.046362 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.052731 (+ 6369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.052739 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.054755 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.054769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.055923 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.055932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.056114 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.056740 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.056761 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.059264 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.063377 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.063385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.063402 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.064077 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.064558 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.064563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.064567 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.064803 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.065476 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.065484 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.066008 (+ 524us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.066320 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.066816 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59547,"mutex_wait_us":34,"spinlock_wait_cycles":2560,"thread_start_us":70346,"threads_started":1}
I20260925 15:07:41.070739 12188 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:41.070785 12189 heartbeater.cc:347] Connected to a master server at 127.4.205.253:32803
I20260925 15:07:41.070807 12187 heartbeater.cc:347] Connected to a master server at 127.4.205.252:33555
I20260925 15:07:41.071056 12188 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.071059 12187 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.071062 12189 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.072022 12188 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
I20260925 15:07:41.072052 12189 heartbeater.cc:511] Master 127.4.205.253:32803 requested a full tablet report, sending...
I20260925 15:07:41.072077 12187 heartbeater.cc:511] Master 127.4.205.252:33555 requested a full tablet report, sending...
I20260925 15:07:41.073761 11074 ts_manager.cc:194] Re-registered known tserver with Master: f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:41.073905 11861 ts_manager.cc:194] Registered new tserver with Master: f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:41.073912 11673 ts_manager.cc:194] Registered new tserver with Master: f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:41.074838 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:46303
I20260925 15:07:41.135515 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11402
I20260925 15:07:41.150898 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:32937
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:37321
--webserver_port=42403
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:43379
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:41.449954 12040 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
W20260925 15:07:41.467077 12200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:41.467334 12200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:41.467499 12200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:41.474470 12200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:41.474557 12200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:41.474747 12200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:41.483709 12200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:43379
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:32937
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=42403
--tserver_master_addrs=127.4.205.254:37321,127.4.205.253:32803,127.4.205.252:33555
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:41.484864 12200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:41.486122 12200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:41.498426 12205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:41.498600 12206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:41.498807 12208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:41.499491 12200 server_base.cc:1034] running on GCE node
I20260925 15:07:41.500023 12200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:41.500855 12200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:41.502063 12200 hybrid_clock.cc:648] HybridClock initialized: now 1790348861502025 us; error 50 us; skew 500 ppm
I20260925 15:07:41.502362 12200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:41.505805 12200 init.cc:377] Logged in from keytab as oryx/127.4.205.195@KRBTEST.COM (short username oryx)
I20260925 15:07:41.507025 12200 webserver.cc:460] Webserver started at http://127.4.205.195:42403/ using document root <none> and password file <none>
I20260925 15:07:41.507666 12200 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:41.507817 12200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:41.513125 12200 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:41.515642 12215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:41.516844 12200 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:07:41.517009 12200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
format_stamp: "Formatted at 2026-09-25 15:07:38 on dist-test-slave-64fx"
server_key: "dcbbd767bc14e0db3bdfe29c9cc18ccf"
server_key_iv: "a97092a40873293dfb853fedfed6f31c"
server_key_version: "encryptionkey@0"
I20260925 15:07:41.517947 12200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:41.545897 12200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:41.550037 12200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:41.550302 12200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:41.551205 12200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:41.552412 12200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:41.552526 12200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:41.552644 12200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:41.552731 12200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:41.583411 12200 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:32937
I20260925 15:07:41.583482 12327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:32937 every 8 connection(s)
I20260925 15:07:41.584730 12200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:41.588888 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12200
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.253@KRBTEST.COM
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
Sep 25 15:07:41 dist-test-slave-64fx krb5kdc[11029](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348861, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.252@KRBTEST.COM
I20260925 15:07:41.626802 12334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.587408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:58317 (local address 127.4.205.253:32803)
0925 15:07:41.599681 (+ 12273us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.599691 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.600572 (+ 881us) server_negotiation.cc:407] Connection header received
0925 15:07:41.604467 (+ 3895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.604472 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.604547 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.604645 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.607404 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.608468 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.611391 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.612166 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.612268 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.617581 (+ 5313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.617589 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.617597 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.617641 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.621388 (+ 3747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.622364 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.622372 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.622378 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.622550 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.623052 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.623059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.623064 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.623450 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.623608 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.625867 (+ 2259us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.626148 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12125,"thread_start_us":12270,"threads_started":1}
I20260925 15:07:41.628639 12333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.587449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:38413 (local address 127.4.205.254:37321)
0925 15:07:41.599683 (+ 12234us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.599692 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.601095 (+ 1403us) server_negotiation.cc:407] Connection header received
0925 15:07:41.604227 (+ 3132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.604231 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.604310 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.604457 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.606962 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.607968 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.611013 (+ 3045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.611603 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.611725 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.617542 (+ 5817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.617552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.617588 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.617637 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.623218 (+ 5581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.624358 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.624364 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.624370 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.624560 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.626083 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.626088 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.626091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.626436 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.626598 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.627773 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.628097 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12086,"thread_start_us":12241,"threads_started":1}
I20260925 15:07:41.630093 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.587292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:40127 (local address 127.4.205.252:33555)
0925 15:07:41.600132 (+ 12840us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.600140 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.604938 (+ 4798us) server_negotiation.cc:407] Connection header received
0925 15:07:41.605489 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.605498 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.605621 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.605696 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.607374 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.608427 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.610550 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.611173 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.611252 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.621862 (+ 10610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.621870 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.621875 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.622302 (+ 427us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.626001 (+ 3699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.627156 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.627161 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.627164 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.627353 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.628125 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.628132 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.628137 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.628464 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.628594 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.629474 (+ 880us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.629755 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12698,"thread_start_us":12617,"threads_started":1}
I20260925 15:07:41.631316 12337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.591085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:38413)
0925 15:07:41.600991 (+ 9906us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.601007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.603913 (+ 2906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.604910 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.604924 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.605901 (+ 977us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.606787 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.606815 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.608257 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.608262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.610872 (+ 2610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.610885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.612069 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.612077 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.612587 (+ 510us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.613361 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.613388 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.617326 (+ 3938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.623670 (+ 6344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.623677 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.623682 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.624238 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.625297 (+ 1059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.625305 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.625311 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.625584 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.627178 (+ 1594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.627186 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.627421 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.629122 (+ 1701us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.629821 (+ 699us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9789,"mutex_wait_us":950,"spinlock_wait_cycles":14080,"thread_start_us":9658,"threads_started":1}
I20260925 15:07:41.631444 12338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.588152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.4.205.195:58317)
0925 15:07:41.600386 (+ 12234us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.600454 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.604321 (+ 3867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.604910 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.604926 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.605774 (+ 848us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.606787 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.606820 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.609208 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.609213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.611235 (+ 2022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.611250 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.612618 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.612626 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.612864 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.613456 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.613484 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.616543 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.621580 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.621593 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.621611 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.622224 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.622695 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.622701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.622707 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.622924 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.624413 (+ 1489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.624424 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.624891 (+ 467us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.630783 (+ 5892us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.631079 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11928,"mutex_wait_us":33,"thread_start_us":16245,"threads_started":1}
I20260925 15:07:41.633674 12330 heartbeater.cc:347] Connected to a master server at 127.4.205.253:32803
I20260925 15:07:41.633759 12329 heartbeater.cc:347] Connected to a master server at 127.4.205.254:37321
I20260925 15:07:41.633940 12329 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.633939 12330 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.634537 12336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.591546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.4.205.195:40127)
0925 15:07:41.604831 (+ 13285us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.604854 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.605038 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.605945 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.605951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.606702 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:41.607215 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.607227 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.608590 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.608600 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.610402 (+ 1802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.610421 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.611526 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.611532 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.611796 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.612532 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:41.612561 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:41.621367 (+ 8806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:41.626144 (+ 4777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.626151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.626157 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.626597 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.627523 (+ 926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:41.627528 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:41.627532 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:41.627695 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:41.628689 (+ 994us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:41.628693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:41.628854 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.633849 (+ 4995us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.634121 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13149,"mutex_wait_us":217,"thread_start_us":8428,"threads_started":1}
I20260925 15:07:41.634932 12330 heartbeater.cc:511] Master 127.4.205.253:32803 requested a full tablet report, sending...
I20260925 15:07:41.635347 12328 heartbeater.cc:347] Connected to a master server at 127.4.205.252:33555
I20260925 15:07:41.635422 12329 heartbeater.cc:511] Master 127.4.205.254:37321 requested a full tablet report, sending...
I20260925 15:07:41.635483 12328 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:41.636193 12328 heartbeater.cc:511] Master 127.4.205.252:33555 requested a full tablet report, sending...
I20260925 15:07:41.636622 11074 ts_manager.cc:194] Re-registered known tserver with Master: dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:41.636852 11673 ts_manager.cc:194] Registered new tserver with Master: dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:41.637300 11861 ts_manager.cc:194] Registered new tserver with Master: dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:41.637775 11074 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:38413
I20260925 15:07:41.645417 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.608890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.4.205.253:32803)
0925 15:07:41.624855 (+ 15965us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.624860 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.624877 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:41.624926 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.624930 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.624991 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.628967 (+ 3976us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.629388 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.630840 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.633663 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.634454 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.635961 (+ 1507us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.638974 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.638982 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.638986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.639019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.642292 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.643131 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.643137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.643142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.643314 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.643743 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.643747 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.643750 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.644117 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.644642 (+ 525us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.644797 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.645007 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15837,"thread_start_us":15235,"threads_started":1}
I20260925 15:07:41.651726 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.614140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.4.205.252:33555)
0925 15:07:41.630370 (+ 16230us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.630376 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.631038 (+ 662us) server_negotiation.cc:407] Connection header received
0925 15:07:41.631084 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.631088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.631176 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.631257 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.638086 (+ 6829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.639169 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.642038 (+ 2869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.642710 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.642840 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.646149 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.646156 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.646164 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.646205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.649013 (+ 2808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.649906 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.649909 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.649911 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.650024 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.650408 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.650411 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.650413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.650648 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.650917 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.651188 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.651394 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16139,"thread_start_us":23697,"threads_started":1}
I20260925 15:07:41.653182 12345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.609287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45356 (local address 127.4.205.254:37321)
0925 15:07:41.623965 (+ 14678us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.623972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.635067 (+ 11095us) server_negotiation.cc:407] Connection header received
0925 15:07:41.635122 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.635127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.635218 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.635329 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:41.640258 (+ 4929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.641421 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.642916 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.643604 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.643754 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.647224 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:41.647229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:41.647232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:41.647259 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:41.650630 (+ 3371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.651362 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.651367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.651371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.651505 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:41.651874 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:41.651881 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:41.651886 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:41.652125 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:41.652364 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.652645 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.652849 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14540,"thread_start_us":14383,"threads_started":1}
I20260925 15:07:41.656566 11074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45356:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:41.663305 11074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:41.693946 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.679960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:32937 (local address 127.0.0.1:50180)
0925 15:07:41.680698 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.680716 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.680823 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.683657 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.683665 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.683698 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:41.684128 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.684143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.687100 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.687107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.689807 (+ 2700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.689826 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.692854 (+ 3028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.692860 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.693046 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.693097 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.693206 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.693437 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":255,"threads_started":1}
I20260925 15:07:41.693964 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.680331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50180 (local address 127.4.205.195:32937)
0925 15:07:41.682176 (+ 1845us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.682189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.682229 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:07:41.683082 (+ 853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.683095 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.683421 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.683640 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:41.684315 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.686953 (+ 2638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.690954 (+ 4001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.692680 (+ 1726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.692829 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.693081 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.693308 (+ 227us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.693517 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1627,"thread_start_us":1438,"threads_started":1}
I20260925 15:07:41.699162 12263 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 (DEFAULT_TABLE table=test-table [id=b07e16e4e4744a258bcf436e8cc6d252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:41.700672 12359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.680759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:33547 (local address 127.0.0.1:46544)
0925 15:07:41.681838 (+ 1079us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.681859 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.681955 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.691369 (+ 9414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.691377 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.691410 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:41.691848 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.691861 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.695982 (+ 4121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.695991 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.698694 (+ 2703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.698713 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.700111 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.700117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.700278 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.700318 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.700376 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.700458 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":967,"thread_start_us":18217,"threads_started":1}
I20260925 15:07:41.701596 12263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4956ed89a44b328414052fb9c41ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:41.701606 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.681265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46544 (local address 127.4.205.194:33547)
0925 15:07:41.690816 (+ 9551us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.690825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.690852 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:07:41.690946 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.690952 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.691170 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.691362 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:41.692077 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.695797 (+ 3720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.698921 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.699889 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.700896 (+ 1007us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.701070 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.701142 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.701295 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9401,"thread_start_us":9279,"threads_started":1}
I20260925 15:07:41.702085 12356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.680811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:39305 (local address 127.0.0.1:56224)
0925 15:07:41.694178 (+ 13367us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.694193 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.694269 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.695029 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.695034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.695062 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:41.695225 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.695235 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.697603 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.697607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.699515 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.699528 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.701201 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.701207 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.701404 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.701436 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.701534 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.701873 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13273,"thread_start_us":281,"threads_started":1}
I20260925 15:07:41.702091 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.681106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56224 (local address 127.4.205.193:39305)
0925 15:07:41.694235 (+ 13129us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.694241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.694263 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:07:41.694466 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.694474 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.694654 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.695406 (+ 752us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:41.695645 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.697458 (+ 1813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.699712 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.701054 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.701186 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.701428 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.701656 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.701798 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12988,"thread_start_us":12662,"threads_started":1}
I20260925 15:07:41.705147 12122 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 (DEFAULT_TABLE table=test-table [id=b07e16e4e4744a258bcf436e8cc6d252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:41.705137 11974 tablet_service.cc:1467] Processing CreateTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 (DEFAULT_TABLE table=test-table [id=b07e16e4e4744a258bcf436e8cc6d252]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:41.706331 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4956ed89a44b328414052fb9c41ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:41.706331 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f4956ed89a44b328414052fb9c41ae4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:41.713481 12362 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Bootstrap starting.
I20260925 15:07:41.716269 12362 tablet_bootstrap.cc:654] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:41.717888 12362 log.cc:824] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:41.720805 12362 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: No bootstrap required, opened a new log
I20260925 15:07:41.721076 12362 ts_tablet_manager.cc:1397] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:07:41.721385 12364 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Bootstrap starting.
I20260925 15:07:41.721680 12365 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Bootstrap starting.
I20260925 15:07:41.726081 12365 tablet_bootstrap.cc:654] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:41.727113 12365 log.cc:824] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:41.728508 12364 tablet_bootstrap.cc:654] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:41.729594 12364 log.cc:824] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:41.729998 12365 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: No bootstrap required, opened a new log
I20260925 15:07:41.730249 12365 ts_tablet_manager.cc:1397] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:07:41.732168 12362 raft_consensus.cc:348] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.732617 12362 raft_consensus.cc:374] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:41.732735 12364 tablet_bootstrap.cc:492] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: No bootstrap required, opened a new log
I20260925 15:07:41.732724 12362 raft_consensus.cc:729] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc8f5b0e9f845b899c94d3e502fece7, State: Initialized, Role: FOLLOWER
I20260925 15:07:41.732964 12364 ts_tablet_manager.cc:1397] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260925 15:07:41.733290 12362 consensus_queue.cc:260] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.734872 12362 ts_tablet_manager.cc:1428] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Time spent starting tablet: real 0.014s user 0.002s sys 0.012s
I20260925 15:07:41.735113 12329 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
I20260925 15:07:41.739202 12365 raft_consensus.cc:348] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.739603 12365 raft_consensus.cc:374] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:41.739881 12365 raft_consensus.cc:729] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8987d797c4c14d8f80a376be1d245505, State: Initialized, Role: FOLLOWER
I20260925 15:07:41.740550 12365 consensus_queue.cc:260] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.742640 12365 ts_tablet_manager.cc:1428] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Time spent starting tablet: real 0.012s user 0.010s sys 0.001s
I20260925 15:07:41.745807 12364 raft_consensus.cc:348] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.746124 12364 raft_consensus.cc:374] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:41.746225 12364 raft_consensus.cc:729] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a1d68ac3bc4357a6f81b46eeb659f1, State: Initialized, Role: FOLLOWER
I20260925 15:07:41.746771 12364 consensus_queue.cc:260] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.747637 12188 heartbeater.cc:503] Master 127.4.205.254:37321 was elected leader, sending a full tablet report...
I20260925 15:07:41.748247 12364 ts_tablet_manager.cc:1428] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Time spent starting tablet: real 0.015s user 0.004s sys 0.010s
W20260925 15:07:41.748594 12194 tablet.cc:2369] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:41.851140 12331 tablet.cc:2369] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:41.885934 12369 raft_consensus.cc:482] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:41.886241 12369 raft_consensus.cc:504] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.887856 12369 leader_election.cc:288] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937), f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:41.895258 12360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.888243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:39609 (local address 127.4.205.194:33547)
0925 15:07:41.888491 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.888499 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.889276 (+ 777us) server_negotiation.cc:407] Connection header received
0925 15:07:41.889454 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.889458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.889516 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.889598 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:41.890371 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.891130 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.893084 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.894307 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.894472 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.894551 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.894895 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.895025 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:07:41.895434 12372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.888071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:33547 (local address 127.4.205.193:39609)
0925 15:07:41.889142 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.889158 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.889309 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.889646 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.889650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.889668 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:41.890181 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.890208 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.891422 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.891427 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.892899 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.892909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.894512 (+ 1603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.894518 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.894711 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.894745 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.894841 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.894973 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":38,"thread_start_us":620,"threads_started":1}
I20260925 15:07:41.895644 12371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.888051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:32937 (local address 127.4.205.193:42833)
0925 15:07:41.888993 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:41.889008 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:41.889130 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:41.889712 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:41.889715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:41.889734 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:41.890135 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.890143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.891652 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.891656 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.893046 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:41.893062 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.894603 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.894609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:41.894767 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.894806 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:07:41.894880 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:07:41.894981 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"mutex_wait_us":17,"thread_start_us":654,"threads_started":1}
I20260925 15:07:41.895749 12357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.888126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42833 (local address 127.4.205.195:32937)
0925 15:07:41.888353 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.888362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.889095 (+ 733us) server_negotiation.cc:407] Connection header received
0925 15:07:41.889265 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.889270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.889344 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.889428 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:41.890268 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.891521 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.893209 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.894425 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.894550 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.894628 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.895212 (+ 584us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.895440 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:07:41.896711 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4956ed89a44b328414052fb9c41ae4" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" is_pre_election: true
I20260925 15:07:41.896711 12283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4956ed89a44b328414052fb9c41ae4" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" is_pre_election: true
I20260925 15:07:41.897248 12283 raft_consensus.cc:2454] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 0.
I20260925 15:07:41.897248 12142 raft_consensus.cc:2454] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 0.
I20260925 15:07:41.897913 11929 leader_election.cc:302] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8987d797c4c14d8f80a376be1d245505, dcc8f5b0e9f845b899c94d3e502fece7; no voters:
I20260925 15:07:41.898226 12369 raft_consensus.cc:2781] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:41.898396 12369 raft_consensus.cc:482] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:41.898504 12369 raft_consensus.cc:3037] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:41.900787 12369 raft_consensus.cc:504] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.901371 12369 leader_election.cc:288] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 election: Requested vote from peers dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937), f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547)
I20260925 15:07:41.901736 12283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4956ed89a44b328414052fb9c41ae4" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
I20260925 15:07:41.901774 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f4956ed89a44b328414052fb9c41ae4" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
I20260925 15:07:41.901932 12283 raft_consensus.cc:3037] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:41.901944 12142 raft_consensus.cc:3037] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:41.904273 12142 raft_consensus.cc:2454] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 1.
I20260925 15:07:41.904300 12283 raft_consensus.cc:2454] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 1.
I20260925 15:07:41.904657 11927 leader_election.cc:302] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8987d797c4c14d8f80a376be1d245505, f2a1d68ac3bc4357a6f81b46eeb659f1; no voters:
I20260925 15:07:41.904935 12369 raft_consensus.cc:2781] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:41.905339 12369 raft_consensus.cc:686] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 1 LEADER]: Becoming Leader. State: Replica: 8987d797c4c14d8f80a376be1d245505, State: Running, Role: LEADER
I20260925 15:07:41.905843 12369 consensus_queue.cc:237] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:41.910612 11073 catalog_manager.cc:5495] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8987d797c4c14d8f80a376be1d245505 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "8987d797c4c14d8f80a376be1d245505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } health_report { overall_health: HEALTHY } } }
W20260925 15:07:41.931960 12042 tablet.cc:2369] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:41.980669 12361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:41.976263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56236 (local address 127.4.205.193:39305)
0925 15:07:41.976434 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:41.976439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:41.976454 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:41.976530 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:41.976533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:41.976607 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:41.976722 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:41.977012 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.977691 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.978949 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:41.979338 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:41.979422 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:41.979800 (+ 378us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:41.980255 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:41.980355 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:41.980439 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:07:41.980492 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:07:41.988577 12142 raft_consensus.cc:1261] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Refusing update from remote peer 8987d797c4c14d8f80a376be1d245505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:41.988593 12283 raft_consensus.cc:1261] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Refusing update from remote peer 8987d797c4c14d8f80a376be1d245505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:41.989519 12373 consensus_queue.cc:1035] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:41.990021 12369 consensus_queue.cc:1035] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:42.008440 12378 mvcc.cc:205] Tried to move back new op lower bound from 7333268938696138752 to 7333268938369953792. Current Snapshot: MvccSnapshot[applied={T|T < 7333268938696138752}]
I20260925 15:07:42.031435 11073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:42.031742 11073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45356:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:42.038769 11073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fb955e793d8d4980b96753d75ce7903f: Sending DeleteTablet for 3 replicas of tablet 5f4956ed89a44b328414052fb9c41ae4
I20260925 15:07:42.039914 12263 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:42 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:50180
I20260925 15:07:42.040340 11974 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:42 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:56224
I20260925 15:07:42.040617 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f4956ed89a44b328414052fb9c41ae4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:42 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:46544
I20260925 15:07:42.052793 12387 tablet_replica.cc:330] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: stopping tablet replica
I20260925 15:07:42.053313 12387 raft_consensus.cc:2229] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:42.055327 12387 raft_consensus.cc:2258] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:42.057324 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11042
I20260925 15:07:42.058408 11104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:42.059965 12385 tablet_replica.cc:330] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: stopping tablet replica
I20260925 15:07:42.060784 12387 ts_tablet_manager.cc:1905] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:42.061921 12385 raft_consensus.cc:2229] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:42.063946 12386 tablet_replica.cc:330] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: stopping tablet replica
I20260925 15:07:42.064443 12385 raft_consensus.cc:2258] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:42.064457 12386 raft_consensus.cc:2229] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:42.065842 12386 raft_consensus.cc:2258] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:42.067500 12385 ts_tablet_manager.cc:1905] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:42.071240 12386 ts_tablet_manager.cc:1905] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:42.078666 12387 ts_tablet_manager.cc:1918] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:42.078841 12387 log.cc:1196] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/5f4956ed89a44b328414052fb9c41ae4
I20260925 15:07:42.079423 12387 ts_tablet_manager.cc:1939] T 5f4956ed89a44b328414052fb9c41ae4 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting consensus metadata
I20260925 15:07:42.083324 12386 ts_tablet_manager.cc:1918] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:42.083492 12386 log.cc:1196] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/5f4956ed89a44b328414052fb9c41ae4
I20260925 15:07:42.084086 12386 ts_tablet_manager.cc:1939] T 5f4956ed89a44b328414052fb9c41ae4 P 8987d797c4c14d8f80a376be1d245505: Deleting consensus metadata
I20260925 15:07:42.086066 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.042399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.4.205.253:32803)
0925 15:07:42.042566 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:42.042573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:42.056008 (+ 13435us) server_negotiation.cc:407] Connection header received
0925 15:07:42.056843 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:42.056850 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:42.056930 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:42.057005 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:42.062159 (+ 5154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.063203 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.067688 (+ 4485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.068329 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.068423 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:42.078969 (+ 10546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:42.078979 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:42.078986 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:42.079036 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:42.082669 (+ 3633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.083725 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.083732 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.083738 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.083875 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.084382 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.084386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.084389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.084733 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:42.084956 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:42.085526 (+ 570us) server_negotiation.cc:299] Negotiation successful
0925 15:07:42.085775 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:42.089241 12385 ts_tablet_manager.cc:1918] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:42.089466 12385 log.cc:1196] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/5f4956ed89a44b328414052fb9c41ae4
I20260925 15:07:42.090021 12385 ts_tablet_manager.cc:1939] T 5f4956ed89a44b328414052fb9c41ae4 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting consensus metadata
I20260925 15:07:42.100461 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.056165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50822 (local address 127.4.205.252:33555)
0925 15:07:42.056372 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:42.056380 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:42.087213 (+ 30833us) server_negotiation.cc:407] Connection header received
0925 15:07:42.087463 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:42.087469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:42.087551 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:42.087623 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:42.089153 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.090407 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.092454 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.092867 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.093012 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:42.095371 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:42.095376 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:42.095379 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:42.095410 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:42.097964 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.098560 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.098568 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.098574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.098699 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.099084 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.099090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.099094 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.099341 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:42.099597 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:42.099889 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:07:42.100095 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:07:42.251713 11059 catalog_manager.cc:4841] TS 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305): tablet 5f4956ed89a44b328414052fb9c41ae4 (table test-table [id=b07e16e4e4744a258bcf436e8cc6d252]) successfully deleted
I20260925 15:07:42.253208 11060 catalog_manager.cc:4841] TS dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937): tablet 5f4956ed89a44b328414052fb9c41ae4 (table test-table [id=b07e16e4e4744a258bcf436e8cc6d252]) successfully deleted
I20260925 15:07:42.254155 11058 catalog_manager.cc:4841] TS f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547): tablet 5f4956ed89a44b328414052fb9c41ae4 (table test-table [id=b07e16e4e4744a258bcf436e8cc6d252]) successfully deleted
W20260925 15:07:42.254349 11058 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.205.194:33547, user_credentials={real_user=slave}} blocked reactor thread for 173069us
I20260925 15:07:42.254912 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11042
I20260925 15:07:42.307334 12343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.279777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.4.205.253:32803)
0925 15:07:42.280005 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:42.280011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:42.291592 (+ 11581us) server_negotiation.cc:407] Connection header received
0925 15:07:42.291839 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:42.291845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:42.291936 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:42.292021 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:42.293843 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.294846 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.296667 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.297288 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.297384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:42.300857 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:42.300863 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:42.300869 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:42.300906 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:42.304444 (+ 3538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.305178 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.305184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.305189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.305312 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.305723 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.305728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.305732 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.306068 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:42.306273 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:42.306777 (+ 504us) server_negotiation.cc:299] Negotiation successful
0925 15:07:42.307002 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 15:07:42.309926 12332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:42.280161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50826 (local address 127.4.205.252:33555)
0925 15:07:42.280381 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:42.280389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:42.291992 (+ 11603us) server_negotiation.cc:407] Connection header received
0925 15:07:42.292135 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:42.292140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:42.292220 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:42.292312 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:42.294560 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.295501 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.297355 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:42.298021 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:42.298132 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:42.303212 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:42.303218 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:42.303222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:42.303266 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:42.306546 (+ 3280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.307302 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.307308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.307313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.307598 (+ 285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:42.308196 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:42.308201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:42.308204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:42.308565 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:42.308801 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:42.309355 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 15:07:42.309605 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:07:43.016036 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.015226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:33219)
0925 15:07:43.015671 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.015811 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":236,"threads_started":1}
W20260925 15:07:43.016389 12040 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:37321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.024674 12398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.023993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:48891)
0925 15:07:43.024401 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.024522 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":198,"threads_started":1}
W20260925 15:07:43.025053 12329 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:37321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.027848 12399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.027168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:39883)
0925 15:07:43.027598 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.027689 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":229,"threads_started":1}
W20260925 15:07:43.028291 12188 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:37321 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.538058 12400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb955e793d8d4980b96753d75ce7903f)
I20260925 15:07:43.538362 12400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } }
I20260925 15:07:43.538983 12401 raft_consensus.cc:482] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader fb955e793d8d4980b96753d75ce7903f)
I20260925 15:07:43.539222 12401 raft_consensus.cc:504] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } }
I20260925 15:07:43.539863 12400 leader_election.cc:288] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321), f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253:32803)
I20260925 15:07:43.540647 12401 leader_election.cc:288] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321), 1e46d6be33c04079919a8a6cb048baa4 (127.4.205.252:33555)
I20260925 15:07:43.553272 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.540146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52834)
0925 15:07:43.552679 (+ 12533us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.552938 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12380,"thread_start_us":10925,"threads_started":1}
W20260925 15:07:43.553581 11845 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.559262 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.550923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52844)
0925 15:07:43.558668 (+ 7745us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.558906 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7628,"thread_start_us":2837,"threads_started":1}
W20260925 15:07:43.559607 11657 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.560839 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.560459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52850)
0925 15:07:43.560619 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.560719 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 15:07:43.561046 11845 leader_election.cc:334] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321): Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.561132 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.560789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52854)
0925 15:07:43.560901 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.561008 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 15:07:43.561266 12404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.540234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57698 (local address 127.4.205.253:32803)
0925 15:07:43.550906 (+ 10672us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:43.550914 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:43.551665 (+ 751us) server_negotiation.cc:407] Connection header received
0925 15:07:43.551753 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:43.551761 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:43.551873 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:43.552001 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:43.552944 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.554351 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.557821 (+ 3470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.559739 (+ 1918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.559884 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.559986 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:43.560731 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 15:07:43.560863 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":10531,"thread_start_us":10364,"threads_started":1}
W20260925 15:07:43.561450 11657 leader_election.cc:334] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321): Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.562112 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.540091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:32803 (local address 127.0.0.1:57698)
0925 15:07:43.551327 (+ 11236us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.551352 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:43.551531 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:43.552220 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:43.552228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:43.552283 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:43.552691 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.552707 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.554559 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.554567 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.557474 (+ 2907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.557493 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.559949 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.559955 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.560162 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.560209 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 15:07:43.561481 (+ 1272us) client_negotiation.cc:240] Negotiation successful
0925 15:07:43.561696 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11101,"mutex_wait_us":4,"thread_start_us":13299,"threads_started":1}
I20260925 15:07:43.562705 11682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e46d6be33c04079919a8a6cb048baa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" is_pre_election: true
I20260925 15:07:43.563063 11682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e46d6be33c04079919a8a6cb048baa4 in term 1.
I20260925 15:07:43.563686 12406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.541672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52816 (local address 127.4.205.252:33555)
0925 15:07:43.550719 (+ 9047us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:43.550726 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:43.554102 (+ 3376us) server_negotiation.cc:407] Connection header received
0925 15:07:43.554363 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:43.554370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:43.554477 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:43.554557 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:43.555659 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.557420 (+ 1761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.561590 (+ 4170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.562701 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.562800 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.562871 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:43.563273 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:07:43.563419 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":8906,"thread_start_us":8765,"threads_started":1}
I20260925 15:07:43.563740 12407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.540817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:33555 (local address 127.0.0.1:52816)
0925 15:07:43.553990 (+ 13173us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.554012 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:43.554200 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:43.554914 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:43.554921 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:43.554953 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:43.555402 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.555415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.557725 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.557733 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.561354 (+ 3621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.561371 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.562874 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.562879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.563088 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.563120 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:07:43.563252 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 15:07:43.563371 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13041,"mutex_wait_us":387,"thread_start_us":17586,"threads_started":1}
I20260925 15:07:43.563619 11847 leader_election.cc:302] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e46d6be33c04079919a8a6cb048baa4, f02e13a1c88741e3b3d0923b85e2f9cf; no voters: fb955e793d8d4980b96753d75ce7903f
I20260925 15:07:43.564086 12400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 15:07:43.564213 12400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader fb955e793d8d4980b96753d75ce7903f)
I20260925 15:07:43.564311 12400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:07:43.564344 11871 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1e46d6be33c04079919a8a6cb048baa4" is_pre_election: true
I20260925 15:07:43.566360 12400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } }
I20260925 15:07:43.566807 11871 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f02e13a1c88741e3b3d0923b85e2f9cf in current term 2: Already voted for candidate 1e46d6be33c04079919a8a6cb048baa4 in this term.
I20260925 15:07:43.567082 12400 leader_election.cc:288] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 election: Requested vote from peers fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321), f02e13a1c88741e3b3d0923b85e2f9cf (127.4.205.253:32803)
I20260925 15:07:43.567883 11658 leader_election.cc:302] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f02e13a1c88741e3b3d0923b85e2f9cf; no voters: 1e46d6be33c04079919a8a6cb048baa4, fb955e793d8d4980b96753d75ce7903f
I20260925 15:07:43.567950 11682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1e46d6be33c04079919a8a6cb048baa4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf"
I20260925 15:07:43.568133 11682 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:07:43.568720 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.567354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52868)
0925 15:07:43.568499 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.568594 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260925 15:07:43.569684 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.569303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52874)
0925 15:07:43.569447 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.569524 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:07:43.569880 11845 leader_election.cc:334] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321): Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
I20260925 15:07:43.570619 11682 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e46d6be33c04079919a8a6cb048baa4 in term 2.
I20260925 15:07:43.570781 12401 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:07:43.571022 11847 leader_election.cc:302] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1e46d6be33c04079919a8a6cb048baa4, f02e13a1c88741e3b3d0923b85e2f9cf; no voters: fb955e793d8d4980b96753d75ce7903f
I20260925 15:07:43.571301 12400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:07:43.571713 12400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [term 2 LEADER]: Becoming Leader. State: Replica: 1e46d6be33c04079919a8a6cb048baa4, State: Running, Role: LEADER
I20260925 15:07:43.572113 12400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } }
I20260925 15:07:43.573742 12410 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e46d6be33c04079919a8a6cb048baa4. Latest consensus state: current_term: 2 leader_uuid: "1e46d6be33c04079919a8a6cb048baa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:43.574013 12410 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:43.574594 12412 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:43.577884 12412 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:43.578676 12412 catalog_manager.cc:1259] Loaded cluster ID: d168e20da74b422ba6f4e43e61800523
I20260925 15:07:43.578768 12412 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:43.579865 12412 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:43.580606 12412 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Loaded TSK: 0
I20260925 15:07:43.581455 12412 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:07:43.625913 11861 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:43.632910 11861 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:43.636731 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.636293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52890)
0925 15:07:43.636437 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.636558 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:07:43.636902 11682 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [term 2 FOLLOWER]: Refusing update from remote peer 1e46d6be33c04079919a8a6cb048baa4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260925 15:07:43.637802 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.637317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52902)
0925 15:07:43.637504 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.637623 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:07:43.637740 12410 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 15:07:43.638015 11845 consensus_peers.cc:487] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 -> Peer fb955e793d8d4980b96753d75ce7903f (127.4.205.254:37321): Couldn't send request to peer fb955e793d8d4980b96753d75ce7903f. Status: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 15:07:43.640302 12401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e46d6be33c04079919a8a6cb048baa4. Latest consensus state: current_term: 2 leader_uuid: "1e46d6be33c04079919a8a6cb048baa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:43.640584 12401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:43.641947 12400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "1e46d6be33c04079919a8a6cb048baa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:43.642259 12400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4 [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:43.642295 12401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "1e46d6be33c04079919a8a6cb048baa4" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "fb955e793d8d4980b96753d75ce7903f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 37321 } } peers { permanent_uuid: "f02e13a1c88741e3b3d0923b85e2f9cf" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 32803 } attrs { promote: false } } peers { permanent_uuid: "1e46d6be33c04079919a8a6cb048baa4" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 33555 } attrs { promote: false } } }
I20260925 15:07:43.642565 12401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f02e13a1c88741e3b3d0923b85e2f9cf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:07:43.655480 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.649496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:33547 (local address 127.0.0.1:48244)
0925 15:07:43.649664 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.649682 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:43.649776 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:43.650437 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:43.650441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:43.650461 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:43.650625 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.650635 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.651833 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.651840 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.653417 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.653427 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.654773 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.654780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.654970 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.654992 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:07:43.655095 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:07:43.655220 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:07:43.655892 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.649779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48244 (local address 127.4.205.194:33547)
0925 15:07:43.650159 (+ 380us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:43.650166 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:43.650183 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:43.650245 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:43.650250 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:43.650316 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:43.650425 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:43.650739 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.651727 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.653552 (+ 1825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.654601 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.654739 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.654854 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:43.655117 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:07:43.655244 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":447,"threads_started":1}
I20260925 15:07:43.656708 12122 tablet_service.cc:1467] Processing CreateTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 (DEFAULT_TABLE table=test-table [id=ad0be36d53574902b66c64c8e60bff27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:43.657087 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc6bbc0410b48d38bf21a16248ba0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:43.660811 12420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.650674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:39305 (local address 127.0.0.1:49014)
0925 15:07:43.651153 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.651168 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:43.651283 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:43.654191 (+ 2908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:43.654198 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:43.654232 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:43.654656 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.654670 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.656593 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.656596 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.658281 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.658317 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.659957 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.659963 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.660155 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.660206 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:07:43.660277 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:07:43.660480 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":483,"threads_started":1}
I20260925 15:07:43.661829 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.651017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49014 (local address 127.4.205.193:39305)
0925 15:07:43.653870 (+ 2853us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:43.653875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:43.653892 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:43.653950 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:43.653956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:43.654044 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:43.654139 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:43.654902 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.656427 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.658493 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.659710 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.661134 (+ 1424us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.661236 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:43.661303 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:07:43.661502 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2745,"thread_start_us":2643,"threads_started":1}
I20260925 15:07:43.662568 11974 tablet_service.cc:1467] Processing CreateTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 (DEFAULT_TABLE table=test-table [id=ad0be36d53574902b66c64c8e60bff27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:43.663080 11974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc6bbc0410b48d38bf21a16248ba0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:43.663183 12422 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Bootstrap starting.
I20260925 15:07:43.664543 12422 tablet_bootstrap.cc:654] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:43.666776 12422 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: No bootstrap required, opened a new log
I20260925 15:07:43.666870 12422 ts_tablet_manager.cc:1397] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260925 15:07:43.667404 12422 raft_consensus.cc:348] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.667577 12422 raft_consensus.cc:374] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:43.667634 12422 raft_consensus.cc:729] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a1d68ac3bc4357a6f81b46eeb659f1, State: Initialized, Role: FOLLOWER
I20260925 15:07:43.667862 12422 consensus_queue.cc:260] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.668571 12422 ts_tablet_manager.cc:1428] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:07:43.672221 12187 heartbeater.cc:503] Master 127.4.205.252:33555 was elected leader, sending a full tablet report...
I20260925 15:07:43.674252 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.649847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:32937 (local address 127.0.0.1:57040)
0925 15:07:43.650014 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.650029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:43.650113 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:43.666986 (+ 16873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:43.666990 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:43.667015 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:43.667319 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.667346 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.668828 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.668835 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.671081 (+ 2246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:43.671093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.673246 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.673256 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:43.673520 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.673592 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 15:07:43.673681 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:07:43.673873 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 15:07:43.675268 12419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.650127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57040 (local address 127.4.205.195:32937)
0925 15:07:43.666589 (+ 16462us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:43.666595 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:43.666612 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:43.666675 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:43.666680 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:43.666755 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:43.667483 (+ 728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:43.667650 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.668676 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.671278 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:43.672960 (+ 1682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:43.674491 (+ 1531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:43.674637 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:43.674698 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:07:43.674893 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16332,"thread_start_us":16708,"threads_started":1}
I20260925 15:07:43.675946 12263 tablet_service.cc:1467] Processing CreateTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 (DEFAULT_TABLE table=test-table [id=ad0be36d53574902b66c64c8e60bff27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:43.676357 12263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbc6bbc0410b48d38bf21a16248ba0d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:43.676971 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.669076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:39243)
0925 15:07:43.676530 (+ 7454us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.676688 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":7312,"threads_started":1}
I20260925 15:07:43.677196 12426 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Bootstrap starting.
I20260925 15:07:43.678720 12426 tablet_bootstrap.cc:654] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:43.680876 12426 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: No bootstrap required, opened a new log
I20260925 15:07:43.681011 12426 ts_tablet_manager.cc:1397] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 15:07:43.681192 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.680782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:49127)
0925 15:07:43.680945 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.681028 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:07:43.681777 12426 raft_consensus.cc:348] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.682016 12426 raft_consensus.cc:374] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:43.682112 12426 raft_consensus.cc:729] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8987d797c4c14d8f80a376be1d245505, State: Initialized, Role: FOLLOWER
I20260925 15:07:43.682343 12426 consensus_queue.cc:260] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.682691 12428 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Bootstrap starting.
I20260925 15:07:43.683224 12426 ts_tablet_manager.cc:1428] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:07:43.684616 12428 tablet_bootstrap.cc:654] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:43.684963 12039 heartbeater.cc:503] Master 127.4.205.252:33555 was elected leader, sending a full tablet report...
I20260925 15:07:43.685796 12430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.683460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:50923)
0925 15:07:43.685387 (+ 1927us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.685512 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1805,"thread_start_us":1680,"threads_started":1}
I20260925 15:07:43.687610 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.686317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:49439)
0925 15:07:43.687194 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.687386 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":654,"threads_started":1}
I20260925 15:07:43.688330 12428 tablet_bootstrap.cc:492] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: No bootstrap required, opened a new log
I20260925 15:07:43.688484 12428 ts_tablet_manager.cc:1397] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260925 15:07:43.689177 12428 raft_consensus.cc:348] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.689373 12428 raft_consensus.cc:374] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:43.689483 12428 raft_consensus.cc:729] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcc8f5b0e9f845b899c94d3e502fece7, State: Initialized, Role: FOLLOWER
I20260925 15:07:43.689738 12428 consensus_queue.cc:260] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.690333 12428 ts_tablet_manager.cc:1428] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:07:43.691382 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.690611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:42857)
0925 15:07:43.691113 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.691215 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":243,"threads_started":1}
I20260925 15:07:43.692292 12328 heartbeater.cc:503] Master 127.4.205.252:33555 was elected leader, sending a full tablet report...
I20260925 15:07:43.692457 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:43.692151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:59485)
0925 15:07:43.692260 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:43.692343 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:07:43.996547 12429 raft_consensus.cc:482] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:43.996820 12429 raft_consensus.cc:504] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:43.997627 12429 leader_election.cc:288] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547), dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:43.998150 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc6bbc0410b48d38bf21a16248ba0d6" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" is_pre_election: true
I20260925 15:07:43.998221 12283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc6bbc0410b48d38bf21a16248ba0d6" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" is_pre_election: true
I20260925 15:07:43.998391 12142 raft_consensus.cc:2454] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 0.
I20260925 15:07:43.998533 12283 raft_consensus.cc:2454] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 0.
I20260925 15:07:43.998862 11927 leader_election.cc:302] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8987d797c4c14d8f80a376be1d245505, f2a1d68ac3bc4357a6f81b46eeb659f1; no voters:
I20260925 15:07:43.999169 12429 raft_consensus.cc:2781] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:43.999316 12429 raft_consensus.cc:482] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:43.999413 12429 raft_consensus.cc:3037] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:44.001384 12429 raft_consensus.cc:504] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:44.001952 12429 leader_election.cc:288] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 election: Requested vote from peers f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547), dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937)
I20260925 15:07:44.002368 12142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc6bbc0410b48d38bf21a16248ba0d6" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1"
I20260925 15:07:44.002393 12283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbc6bbc0410b48d38bf21a16248ba0d6" candidate_uuid: "8987d797c4c14d8f80a376be1d245505" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dcc8f5b0e9f845b899c94d3e502fece7"
I20260925 15:07:44.002566 12142 raft_consensus.cc:3037] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:44.002586 12283 raft_consensus.cc:3037] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:44.004521 12283 raft_consensus.cc:2454] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 1.
I20260925 15:07:44.004530 12142 raft_consensus.cc:2454] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8987d797c4c14d8f80a376be1d245505 in term 1.
I20260925 15:07:44.004940 11927 leader_election.cc:302] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8987d797c4c14d8f80a376be1d245505, f2a1d68ac3bc4357a6f81b46eeb659f1; no voters:
I20260925 15:07:44.005221 12429 raft_consensus.cc:2781] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:44.005674 12429 raft_consensus.cc:686] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 1 LEADER]: Becoming Leader. State: Replica: 8987d797c4c14d8f80a376be1d245505, State: Running, Role: LEADER
I20260925 15:07:44.005967 12429 consensus_queue.cc:237] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } }
I20260925 15:07:44.006691 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.006231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:44483)
0925 15:07:44.006388 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.006502 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:07:44.008977 11861 catalog_manager.cc:5495] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8987d797c4c14d8f80a376be1d245505 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "8987d797c4c14d8f80a376be1d245505" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8987d797c4c14d8f80a376be1d245505" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39305 } health_report { overall_health: HEALTHY } } }
I20260925 15:07:44.020313 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.019997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52910)
0925 15:07:44.020120 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.020192 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:07:44.021186 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.020863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52916)
0925 15:07:44.021020 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.021081 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:07:44.108952 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.104371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.4.205.193:39305)
0925 15:07:44.104561 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:44.104566 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:44.104584 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:07:44.104760 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:44.104766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:44.104840 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:44.104937 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:44.105313 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.106076 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.107341 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.107756 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.107885 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:44.108279 (+ 394us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:44.108370 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:44.108467 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:44.108616 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:07:44.108688 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:07:44.111819 12142 raft_consensus.cc:1261] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Refusing update from remote peer 8987d797c4c14d8f80a376be1d245505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:44.111863 12283 raft_consensus.cc:1261] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Refusing update from remote peer 8987d797c4c14d8f80a376be1d245505: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:44.112461 12435 consensus_queue.cc:1035] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2a1d68ac3bc4357a6f81b46eeb659f1" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 33547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:44.112808 12429 consensus_queue.cc:1035] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcc8f5b0e9f845b899c94d3e502fece7" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 32937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:44.115042 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.114517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:38461)
0925 15:07:44.114719 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.114800 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260925 15:07:44.115584 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.114940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:33587)
0925 15:07:44.115231 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.115366 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
I20260925 15:07:44.120322 12434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.119876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.195:52603)
0925 15:07:44.120018 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.120121 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:07:44.127379 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.126792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:45979)
0925 15:07:44.126947 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.127042 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:07:44.133138 12432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.132737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.193:38799)
0925 15:07:44.132927 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.133008 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 15:07:44.134728 12425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.134415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.4.205.194:46839)
0925 15:07:44.134548 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.134626 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:07:44.136543 11861 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:44.136911 11861 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:44.141423 11861 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1e46d6be33c04079919a8a6cb048baa4: Sending DeleteTablet for 3 replicas of tablet bbc6bbc0410b48d38bf21a16248ba0d6
I20260925 15:07:44.142206 12122 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:44 UTC) from {username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:48244
I20260925 15:07:44.142499 12263 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:44 UTC) from {username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:57040
I20260925 15:07:44.142638 11974 tablet_service.cc:1514] Processing DeleteTablet for tablet bbc6bbc0410b48d38bf21a16248ba0d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:44 UTC) from {username='oryx', principal='oryx/127.4.205.252@KRBTEST.COM'} at 127.0.0.1:49014
I20260925 15:07:44.144497 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11840
I20260925 15:07:44.151976 12449 tablet_replica.cc:330] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: stopping tablet replica
I20260925 15:07:44.152163 12448 tablet_replica.cc:330] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: stopping tablet replica
I20260925 15:07:44.152426 12448 raft_consensus.cc:2229] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:44.152659 12449 raft_consensus.cc:2229] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:44.152709 12448 raft_consensus.cc:2258] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:44.152938 12449 raft_consensus.cc:2258] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:44.153770 12448 ts_tablet_manager.cc:1905] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:44.153996 12450 tablet_replica.cc:330] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: stopping tablet replica
I20260925 15:07:44.154232 12450 raft_consensus.cc:2229] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:44.154565 12449 ts_tablet_manager.cc:1905] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:44.154690 12450 raft_consensus.cc:2258] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:44.155763 12450 ts_tablet_manager.cc:1905] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:44.160223 12448 ts_tablet_manager.cc:1918] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:44.160414 12448 log.cc:1196] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/bbc6bbc0410b48d38bf21a16248ba0d6
I20260925 15:07:44.161031 12449 ts_tablet_manager.cc:1918] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:44.161186 12448 ts_tablet_manager.cc:1939] T bbc6bbc0410b48d38bf21a16248ba0d6 P f2a1d68ac3bc4357a6f81b46eeb659f1: Deleting consensus metadata
I20260925 15:07:44.161464 12449 log.cc:1196] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/bbc6bbc0410b48d38bf21a16248ba0d6
I20260925 15:07:44.161984 12449 ts_tablet_manager.cc:1939] T bbc6bbc0410b48d38bf21a16248ba0d6 P dcc8f5b0e9f845b899c94d3e502fece7: Deleting consensus metadata
I20260925 15:07:44.162545 11845 catalog_manager.cc:4841] TS f2a1d68ac3bc4357a6f81b46eeb659f1 (127.4.205.194:33547): tablet bbc6bbc0410b48d38bf21a16248ba0d6 (table test-table [id=ad0be36d53574902b66c64c8e60bff27]) successfully deleted
I20260925 15:07:44.163002 12450 ts_tablet_manager.cc:1918] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:44.163031 11847 catalog_manager.cc:4841] TS dcc8f5b0e9f845b899c94d3e502fece7 (127.4.205.195:32937): tablet bbc6bbc0410b48d38bf21a16248ba0d6 (table test-table [id=ad0be36d53574902b66c64c8e60bff27]) successfully deleted
I20260925 15:07:44.163136 12450 log.cc:1196] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/bbc6bbc0410b48d38bf21a16248ba0d6
I20260925 15:07:44.163540 12450 ts_tablet_manager.cc:1939] T bbc6bbc0410b48d38bf21a16248ba0d6 P 8987d797c4c14d8f80a376be1d245505: Deleting consensus metadata
I20260925 15:07:44.164597 11846 catalog_manager.cc:4841] TS 8987d797c4c14d8f80a376be1d245505 (127.4.205.193:39305): tablet bbc6bbc0410b48d38bf21a16248ba0d6 (table test-table [id=ad0be36d53574902b66c64c8e60bff27]) successfully deleted
I20260925 15:07:44.169473 12421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.145752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.4.205.193:39305)
0925 15:07:44.145948 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:44.145957 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:44.155278 (+ 9321us) server_negotiation.cc:407] Connection header received
0925 15:07:44.155537 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:44.155542 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:44.155611 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:44.155689 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:44.157482 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.158584 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.160407 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.161087 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.161181 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:44.164766 (+ 3585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:44.164787 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:44.164802 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:44.164830 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:44.167453 (+ 2623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.168014 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.168021 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.168023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.168136 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.168437 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.168440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.168442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.168737 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:44.168950 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:44.169113 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:07:44.169249 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:07:44.170147 12034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:44.287453 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11840
I20260925 15:07:44.308794 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12051
I20260925 15:07:44.321110 12453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.309452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.4.205.194:33547)
0925 15:07:44.309891 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:44.309895 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:44.309909 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:07:44.309986 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:44.309992 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:44.310065 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:44.310148 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:44.311482 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.312244 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.313330 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.313753 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.313839 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:44.316070 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:44.316084 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:44.316116 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:44.316154 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:44.318728 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.319289 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.319296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.319298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.319414 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.319839 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.319842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.319844 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.320140 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:44.320353 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:44.320544 (+ 191us) server_negotiation.cc:299] Negotiation successful
0925 15:07:44.320696 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":239,"threads_started":1}
I20260925 15:07:44.321831 12182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:44.414938 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12051
I20260925 15:07:44.432512 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12200
I20260925 15:07:44.445288 12455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.433187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57052 (local address 127.4.205.195:32937)
0925 15:07:44.433669 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:44.433673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:44.433686 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:07:44.433735 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:44.433738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:44.433790 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:44.433901 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:44.435282 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.435971 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.437216 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:44.437667 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:44.437795 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:44.440131 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:44.440152 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:44.440167 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:44.440197 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:44.442721 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.443371 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.443386 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.443390 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.443511 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:44.443876 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:44.443883 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:44.443887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:44.444209 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:44.444523 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:44.444788 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:07:44.445003 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":211,"threads_started":1}
I20260925 15:07:44.446043 12323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:44.540076 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12200
I20260925 15:07:44.556154 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11535
I20260925 15:07:44.557264 11703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:44.606535 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.605918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52920)
0925 15:07:44.606291 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.606376 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":189,"threads_started":1}
I20260925 15:07:44.607905 12458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:44.607589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:37321 (local address 127.0.0.1:52930)
0925 15:07:44.607725 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:44.607808 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:37321: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 15:07:44.667976 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11535
I20260925 15:07:44.687978 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 11633
I20260925 15:07:44.688992 11891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:44.804034 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 11633
2026-09-25T15:07:44Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9458 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:44 dist-test-slave-64fx krb5kdc[12463](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:44 dist-test-slave-64fx krb5kdc[12463](info): set up 2 sockets
Sep 25 15:07:44 dist-test-slave-64fx krb5kdc[12463](info): commencing operation
krb5kdc: starting...
W20260925 15:07:45.870451 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:45 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab.
Entry for principal oryx/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab.
I20260925 15:07:46.058861 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35929
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab
--principal=oryx/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:35929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:46.360519 12476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:46.360853 12476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:46.360999 12476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:46.367790 12476 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:46.367899 12476 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 15:07:46.367982 12476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:46.368089 12476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:46.368165 12476 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:46.368247 12476 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:46.377719 12476 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:35929
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.254.keytab
--principal=oryx/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:46.378897 12476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:46.380106 12476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:46.391088 12481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:46.391055 12484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:46.391009 12482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:46.391844 12476 server_base.cc:1034] running on GCE node
I20260925 15:07:46.392575 12476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:46.393821 12476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:46.395053 12476 hybrid_clock.cc:648] HybridClock initialized: now 1790348866394981 us; error 56 us; skew 500 ppm
I20260925 15:07:46.395494 12476 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:46 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:46.399187 12476 init.cc:377] Logged in from keytab as oryx/127.4.205.254@KRBTEST.COM (short username oryx)
I20260925 15:07:46.406855 12476 webserver.cc:460] Webserver started at http://127.4.205.254:40803/ using document root <none> and password file <none>
I20260925 15:07:46.407723 12476 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:46.407837 12476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:46.408155 12476 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:46.411123 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "54741c469c854cc5bc017bfb6375394a"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "7010f3e8ba54a25c901a6c0461980082"
server_key_iv: "99836eae41ce44dddc10ee0d4fb94b78"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.411885 12476 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "54741c469c854cc5bc017bfb6375394a"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "7010f3e8ba54a25c901a6c0461980082"
server_key_iv: "99836eae41ce44dddc10ee0d4fb94b78"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.416719 12476 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:07:46.420022 12491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:46.421275 12476 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:46.421409 12476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "54741c469c854cc5bc017bfb6375394a"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "7010f3e8ba54a25c901a6c0461980082"
server_key_iv: "99836eae41ce44dddc10ee0d4fb94b78"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.421643 12476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:46.435196 12476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:46.438803 12476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:46.439095 12476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:46.458455 12476 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:35929
I20260925 15:07:46.458473 12552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:35929 every 8 connection(s)
I20260925 15:07:46.459693 12476 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:46.463143 12553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:46.466337 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12476
I20260925 15:07:46.466518 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:46.467217 4919 external_mini_cluster.cc:1428] Setting key 5a3ad9c2907e8876ba30462e4bb22aa8
I20260925 15:07:46.474962 12553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Bootstrap starting.
I20260925 15:07:46.478248 12553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:46.479493 12553 log.cc:824] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:07:46 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348865, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:46.482028 12553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: No bootstrap required, opened a new log
I20260925 15:07:46.487929 12556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:46.469047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43660 (local address 127.4.205.254:35929)
0925 15:07:46.469796 (+ 749us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:46.469806 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:46.469841 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:07:46.471359 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:46.471378 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:46.471726 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:46.472042 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:46.473645 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:46.474779 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:46.476287 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:46.476902 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:46.477026 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:46.481536 (+ 4510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:46.481543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:46.481599 (+ 56us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:46.481646 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:46.484350 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:46.484973 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:46.484986 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:46.484991 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:46.485131 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:46.485493 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:46.485500 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:46.485507 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:46.486007 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:46.486294 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:46.486681 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 15:07:46.487067 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":354,"threads_started":1}
I20260925 15:07:46.489184 12553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } }
I20260925 15:07:46.489635 12553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:46.489739 12553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54741c469c854cc5bc017bfb6375394a, State: Initialized, Role: FOLLOWER
I20260925 15:07:46.490322 12553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } }
I20260925 15:07:46.490610 12553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:46.490738 12553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:46.490854 12553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:46.492719 12553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } }
I20260925 15:07:46.493201 12553 leader_election.cc:302] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54741c469c854cc5bc017bfb6375394a; no voters:
I20260925 15:07:46.493799 12553 leader_election.cc:288] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:46.493901 12558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:46.494590 12558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [term 1 LEADER]: Becoming Leader. State: Replica: 54741c469c854cc5bc017bfb6375394a, State: Running, Role: LEADER
I20260925 15:07:46.495102 12558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } }
I20260925 15:07:46.495735 12553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:46.498061 12560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54741c469c854cc5bc017bfb6375394a. Latest consensus state: current_term: 1 leader_uuid: "54741c469c854cc5bc017bfb6375394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } } }
I20260925 15:07:46.498445 12560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:46.498956 12559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54741c469c854cc5bc017bfb6375394a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54741c469c854cc5bc017bfb6375394a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35929 } } }
I20260925 15:07:46.499269 12559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:46.500777 12567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:46.505653 12567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:46.513103 12567 catalog_manager.cc:1347] Generated new cluster ID: 3fd78cb3570b4af6a1a6344df58e6e87
I20260925 15:07:46.513247 12567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:46.532847 12567 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:46.533991 12567 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:46.539873 12567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Generated new TSK 0
I20260925 15:07:46.540584 12567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
I20260925 15:07:46.598532 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35929
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:46.923753 12579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:46.924062 12579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:46.924168 12579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:46.931315 12579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:46.931409 12579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:46.931490 12579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:46.931561 12579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:46.940678 12579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:46.941991 12579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:46.943276 12579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:46.956202 12584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:46.956250 12585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:46.956156 12587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:46.962051 12579 server_base.cc:1034] running on GCE node
I20260925 15:07:46.962656 12579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:46.963532 12579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:46.964740 12579 hybrid_clock.cc:648] HybridClock initialized: now 1790348866964679 us; error 35 us; skew 500 ppm
I20260925 15:07:46.965044 12579 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:46 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:46.968163 12579 init.cc:377] Logged in from keytab as oryx/127.4.205.193@KRBTEST.COM (short username oryx)
I20260925 15:07:46.969710 12579 webserver.cc:460] Webserver started at http://127.4.205.193:38427/ using document root <none> and password file <none>
I20260925 15:07:46.970402 12579 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:46.970533 12579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:46.970839 12579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:46.973677 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "ae7b58600bbe4cf38c12cd53518ba26d"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "3af50e9b6a0b961bd7b1120e97e69d55"
server_key_iv: "1cb74f8238b1194a073cdebce635bbfa"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.974426 12579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "ae7b58600bbe4cf38c12cd53518ba26d"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "3af50e9b6a0b961bd7b1120e97e69d55"
server_key_iv: "1cb74f8238b1194a073cdebce635bbfa"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.979157 12579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:07:46.982314 12594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:46.983544 12579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:46.983711 12579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "ae7b58600bbe4cf38c12cd53518ba26d"
format_stamp: "Formatted at 2026-09-25 15:07:46 on dist-test-slave-64fx"
server_key: "3af50e9b6a0b961bd7b1120e97e69d55"
server_key_iv: "1cb74f8238b1194a073cdebce635bbfa"
server_key_version: "encryptionkey@0"
I20260925 15:07:46.983922 12579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:47.004951 12579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:47.009326 12579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:47.009657 12579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:47.012065 12579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:47.012184 12579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:47.012355 12579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:47.012434 12579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:47 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:47.037906 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.012859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42437 (local address 127.4.205.254:35929)
0925 15:07:47.013311 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.013317 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.014702 (+ 1385us) server_negotiation.cc:407] Connection header received
0925 15:07:47.017754 (+ 3052us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.017760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.017864 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.018011 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.020675 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.021430 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.024654 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.025251 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.025400 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.030443 (+ 5043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.030448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.030452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.030514 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.033866 (+ 3352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.034651 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.034656 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.034660 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.034819 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.035478 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.035483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.035487 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.035917 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.036078 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.037070 (+ 992us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.037372 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":229,"threads_started":1}
I20260925 15:07:47.039011 12604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.013609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.193:42437)
0925 15:07:47.014506 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:47.014559 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:47.017379 (+ 2820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:47.018348 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:47.018363 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:47.019371 (+ 1008us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:47.020432 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.020470 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.021634 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.021642 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.024498 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.024517 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.025523 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.025533 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.025953 (+ 420us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.026726 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:47.026772 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:47.030215 (+ 3443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:47.034002 (+ 3787us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.034015 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.034032 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.034503 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.034987 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.034994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.035000 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.035306 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.036073 (+ 767us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:47.036087 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:47.036604 (+ 517us) client_negotiation.cc:769] Sending connection context
0925 15:07:47.036969 (+ 365us) client_negotiation.cc:240] Negotiation successful
0925 15:07:47.037375 (+ 406us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":331,"threads_started":1}
I20260925 15:07:47.057515 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.044669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:35187 (local address 127.4.205.254:35929)
0925 15:07:47.044810 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.044814 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.045471 (+ 657us) server_negotiation.cc:407] Connection header received
0925 15:07:47.045680 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.045692 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.045746 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.045841 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.046717 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.047412 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.048690 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.049149 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.049278 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.051640 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.051647 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.051653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.051707 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.054969 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.055730 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.055733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.055735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.055837 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.056212 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.056219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.056223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.056460 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.056611 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.056966 (+ 355us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.057163 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:07:47.057674 12665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.044775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.193:35187)
0925 15:07:47.045370 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:47.045385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:47.045543 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:47.045871 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:47.045878 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:47.046316 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:47.046584 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.046593 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.047541 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.047548 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.048532 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.048543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.049337 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.049344 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.049603 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.050322 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:47.050341 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:47.051386 (+ 1045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:47.055139 (+ 3753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.055144 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.055148 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.055608 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.055950 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.055957 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.055962 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.056112 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.056573 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:47.056577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:47.056745 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:07:47.056879 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:07:47.057164 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":345,"threads_started":1}
I20260925 15:07:47.067637 12579 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:38829
I20260925 15:07:47.067658 12714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:38829 every 8 connection(s)
I20260925 15:07:47.068815 12579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:47.069819 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12579
I20260925 15:07:47.070014 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:47.070753 4919 external_mini_cluster.cc:1428] Setting key 10df24b14021bc31fd9b3824bdccb77f
I20260925 15:07:47.070785 12715 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35929
I20260925 15:07:47.070994 12715 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:47.071682 12715 heartbeater.cc:511] Master 127.4.205.254:35929 requested a full tablet report, sending...
I20260925 15:07:47.073733 12508 ts_manager.cc:194] Registered new tserver with Master: ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829)
I20260925 15:07:47.075831 12508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:42437
WARNING: no policy specified for oryx/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
I20260925 15:07:47.113834 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35929
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:47.420409 12719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:47.420646 12719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:47.420735 12719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:47.427663 12719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:47.427778 12719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:47.427937 12719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:47.428061 12719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:47.436746 12719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:47.437983 12719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:47.439224 12719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:47.451531 12724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:47.451510 12725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:47.451496 12727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:47.452419 12719 server_base.cc:1034] running on GCE node
I20260925 15:07:47.452988 12719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:47.453889 12719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:47.455120 12719 hybrid_clock.cc:648] HybridClock initialized: now 1790348867455056 us; error 42 us; skew 500 ppm
I20260925 15:07:47.455399 12719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:47 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348867, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:47.458448 12719 init.cc:377] Logged in from keytab as oryx/127.4.205.194@KRBTEST.COM (short username oryx)
I20260925 15:07:47.459900 12719 webserver.cc:460] Webserver started at http://127.4.205.194:44647/ using document root <none> and password file <none>
I20260925 15:07:47.460533 12719 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:47.460614 12719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:47.460942 12719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:47.463687 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "ad0319b5273145b8ba8cf56f74086d1c"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "6b600909c463ac1608759388bdc71d73"
server_key_iv: "e6d09d718a5fe025726f81d248ecd41a"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.464412 12719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "ad0319b5273145b8ba8cf56f74086d1c"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "6b600909c463ac1608759388bdc71d73"
server_key_iv: "e6d09d718a5fe025726f81d248ecd41a"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.469270 12719 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:07:47.472637 12734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:47.473906 12719 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260925 15:07:47.474066 12719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "ad0319b5273145b8ba8cf56f74086d1c"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "6b600909c463ac1608759388bdc71d73"
server_key_iv: "e6d09d718a5fe025726f81d248ecd41a"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.474278 12719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:47.500151 12719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:47.503784 12719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:47.504046 12719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:47.506644 12719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:47.506778 12719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:47.506911 12719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:47.507022 12719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:47 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348867, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:47.529174 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.506844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:33641 (local address 127.4.205.254:35929)
0925 15:07:47.507027 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.507033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.508564 (+ 1531us) server_negotiation.cc:407] Connection header received
0925 15:07:47.510525 (+ 1961us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.510533 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.510620 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.510764 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.512753 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.514063 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.516520 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.516959 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.517080 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.521627 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.521635 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.521643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.521688 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.525033 (+ 3345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.526173 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.526180 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.526185 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.526339 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.526990 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.526996 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.527000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.527368 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.527503 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.528506 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.528766 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:07:47.530997 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.507580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.194:33641)
0925 15:07:47.508392 (+ 812us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:47.508442 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:47.510275 (+ 1833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:47.511012 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:47.511022 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:47.511763 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:47.512572 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.512599 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.514229 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.514237 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.516366 (+ 2129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.516376 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.517183 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.517190 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.517439 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.518220 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:47.518255 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:47.521390 (+ 3135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:47.525213 (+ 3823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.525223 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.525237 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.525999 (+ 762us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.526427 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.526433 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.526444 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.526828 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.527583 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:47.527593 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:47.528059 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 15:07:47.528380 (+ 321us) client_negotiation.cc:240] Negotiation successful
0925 15:07:47.529018 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":339,"threads_started":1}
I20260925 15:07:47.549059 12719 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:37873
I20260925 15:07:47.549065 12853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:37873 every 8 connection(s)
I20260925 15:07:47.550207 12823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.535977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.194:32847)
0925 15:07:47.536854 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:47.536877 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:47.537047 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:47.537413 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:47.537420 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:47.538214 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:47.538599 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.538614 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.539547 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.539553 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.540809 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:47.540819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.541730 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.541737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:47.541913 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.542575 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:47.542590 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:47.543626 (+ 1036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:47.547479 (+ 3853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.547482 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.547484 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.547851 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.548294 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:47.548301 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:47.548307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:47.548461 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:47.549324 (+ 863us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:47.549339 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:47.549476 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:07:47.549667 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:07:47.549880 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":411,"threads_started":1}
I20260925 15:07:47.550302 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:47.535858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:32847 (local address 127.4.205.254:35929)
0925 15:07:47.536041 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:47.536046 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:47.536954 (+ 908us) server_negotiation.cc:407] Connection header received
0925 15:07:47.537171 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:47.537177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:47.537255 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:47.537363 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:47.538761 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.539418 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.540935 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:47.541521 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:47.541679 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:47.543835 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:47.543844 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:47.543852 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:47.543896 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:47.547357 (+ 3461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.547993 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.547998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.548000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.548139 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:47.548641 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:47.548667 (+ 26us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:47.548671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:47.549115 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:47.549294 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:47.549761 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 15:07:47.549987 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:07:47.550530 12719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:47.552834 12854 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35929
I20260925 15:07:47.553061 12854 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:47.553818 12854 heartbeater.cc:511] Master 127.4.205.254:35929 requested a full tablet report, sending...
I20260925 15:07:47.555009 12508 ts_manager.cc:194] Registered new tserver with Master: ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873)
I20260925 15:07:47.555725 12508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:33641
I20260925 15:07:47.560489 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12719
I20260925 15:07:47.560612 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:47.561017 4919 external_mini_cluster.cc:1428] Setting key 414a2323ee49863c225fb9a297ed3759
WARNING: no policy specified for oryx/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
I20260925 15:07:47.600543 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35929
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44745
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:47.905730 12858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:47.905975 12858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:47.906154 12858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:47.913018 12858 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:47.913139 12858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:47.913290 12858 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 15:07:47.913416 12858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:47.922346 12858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44745
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.4.205.254:35929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:47.923472 12858 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:47.924722 12858 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:47.937224 12864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:47.937264 12863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:47.937286 12866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:47.941738 12858 server_base.cc:1034] running on GCE node
I20260925 15:07:47.942350 12858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:47.943256 12858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:47.944499 12858 hybrid_clock.cc:648] HybridClock initialized: now 1790348867944421 us; error 41 us; skew 500 ppm
I20260925 15:07:47.944813 12858 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:47 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348867, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:47.952733 12858 init.cc:377] Logged in from keytab as oryx/127.4.205.195@KRBTEST.COM (short username oryx)
I20260925 15:07:47.954253 12858 webserver.cc:460] Webserver started at http://127.4.205.195:33461/ using document root <none> and password file <none>
I20260925 15:07:47.954993 12858 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:47.955114 12858 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:47.955428 12858 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:47.958335 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "b87541fdc92a478fb1e45b35f01baf8d"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "38f3b5a4c8adcf4ac5abbdef8d589210"
server_key_iv: "85845d7fda39f9a479fca9d7495c1171"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.959118 12858 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "b87541fdc92a478fb1e45b35f01baf8d"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "38f3b5a4c8adcf4ac5abbdef8d589210"
server_key_iv: "85845d7fda39f9a479fca9d7495c1171"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.963830 12858 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:47.966941 12873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:47.968070 12858 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:47.968230 12858 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "b87541fdc92a478fb1e45b35f01baf8d"
format_stamp: "Formatted at 2026-09-25 15:07:47 on dist-test-slave-64fx"
server_key: "38f3b5a4c8adcf4ac5abbdef8d589210"
server_key_iv: "85845d7fda39f9a479fca9d7495c1171"
server_key_version: "encryptionkey@0"
I20260925 15:07:47.968448 12858 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:47.990029 12858 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:47.994854 12858 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:47.995146 12858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:47.998430 12858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:47.998566 12858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 15:07:47.998752 12858 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:47.998842 12858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:07:48 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348867, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:48.029246 12882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:48.000140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.195:59815)
0925 15:07:48.001036 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:48.001070 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:48.004165 (+ 3095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:48.005438 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:48.005450 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:48.006545 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:48.007730 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:48.007758 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.009795 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.009804 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:48.012561 (+ 2757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:48.012582 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.013733 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.013742 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:48.014116 (+ 374us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:48.015063 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:48.015101 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:48.019008 (+ 3907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:48.024061 (+ 5053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:48.024075 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:48.024095 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:48.024735 (+ 640us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:48.025229 (+ 494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:48.025234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:48.025237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:48.025622 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:48.026504 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:48.026515 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:48.026992 (+ 477us) client_negotiation.cc:769] Sending connection context
0925 15:07:48.027307 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 15:07:48.027817 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":746,"threads_started":1}
I20260925 15:07:48.030176 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:48.000814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:59815 (local address 127.4.205.254:35929)
0925 15:07:48.001013 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:48.001019 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:48.001991 (+ 972us) server_negotiation.cc:407] Connection header received
0925 15:07:48.004668 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:48.004674 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:48.004760 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:48.004852 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:48.007980 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.009498 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.012771 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.013444 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.013608 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:48.019319 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:48.019329 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:48.019337 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:48.019392 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:48.023857 (+ 4465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:48.024898 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:48.024904 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:48.024907 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:48.025084 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:48.025783 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:48.025789 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:48.025792 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:48.026247 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:48.026423 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:48.029772 (+ 3349us) server_negotiation.cc:299] Negotiation successful
0925 15:07:48.029949 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:07:48.057782 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:48.043704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:46823 (local address 127.4.205.254:35929)
0925 15:07:48.043882 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:48.043889 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:48.044456 (+ 567us) server_negotiation.cc:407] Connection header received
0925 15:07:48.044651 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:48.044657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:48.044755 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:48.044841 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:48.045713 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.046800 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.048284 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.048800 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.048922 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:48.051379 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:48.051387 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:48.051394 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:48.051434 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:48.054767 (+ 3333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:48.055569 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:48.055576 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:48.055581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:48.055699 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:48.056142 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:48.056147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:48.056151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:48.056496 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:48.056621 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:48.057188 (+ 567us) server_negotiation.cc:299] Negotiation successful
0925 15:07:48.057410 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:07:48.057940 12943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:48.043743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.4.205.195:46823)
0925 15:07:48.044346 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:48.044378 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:48.044526 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:48.044867 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:48.044871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:48.045281 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:48.045543 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:48.045551 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.046905 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.046912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:48.048107 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:48.048126 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.049232 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.049242 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:48.049450 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:48.050491 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:48.050512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:48.051196 (+ 684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:48.054888 (+ 3692us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:48.054896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:48.054903 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:48.055457 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:48.055803 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:48.055810 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:48.055818 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:48.055994 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:48.056702 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:48.056707 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:48.056905 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 15:07:48.057098 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:07:48.057473 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":346,"threads_started":1}
I20260925 15:07:48.066373 12858 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:44947
I20260925 15:07:48.066401 12992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:44947 every 8 connection(s)
I20260925 15:07:48.067469 12858 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:48.069546 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 12858
I20260925 15:07:48.069628 12993 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35929
I20260925 15:07:48.069730 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:48.069790 12993 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:48.070219 4919 external_mini_cluster.cc:1428] Setting key 12d99f8ee287e560ef8197c5a772b83a
I20260925 15:07:48.070299 12993 heartbeater.cc:511] Master 127.4.205.254:35929 requested a full tablet report, sending...
I20260925 15:07:48.071632 12508 ts_manager.cc:194] Registered new tserver with Master: b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:48.072738 12508 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:59815
I20260925 15:07:48.073309 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:07:48.078411 12715 heartbeater.cc:503] Master 127.4.205.254:35929 was elected leader, sending a full tablet report...
Sep 25 15:07:48 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 15:07:48 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:48 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:07:48.557821 12854 heartbeater.cc:503] Master 127.4.205.254:35929 was elected leader, sending a full tablet report...
I20260925 15:07:49.074851 12993 heartbeater.cc:503] Master 127.4.205.254:35929 was elected leader, sending a full tablet report...
W20260925 15:07:49.141381 13001 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.029s user 0.000s sys 0.012s
W20260925 15:07:49.142460 12603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:48.106854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.4.205.254:35929)
0925 15:07:48.107098 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:48.107105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:48.107325 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:07:48.107510 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:48.107515 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:48.107575 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:48.107667 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:48.108855 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.109605 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.110773 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:48.111155 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:48.111276 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.142243 (+1030967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43668: BlockingRecv error: recv got EOF from 127.0.0.1:43668 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 15:07:49 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348868, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:49.159078 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.146234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.4.205.254:35929)
0925 15:07:49.146459 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.146465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.146700 (+ 235us) server_negotiation.cc:407] Connection header received
0925 15:07:49.146914 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.146919 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.147001 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.147096 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:49.148236 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.148974 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.150294 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.150701 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.150839 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.154368 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:49.154373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:49.154376 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:49.154406 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:49.156798 (+ 2392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.157450 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.157454 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.157456 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.157648 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.157975 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.157978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.157980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.158224 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:49.158405 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.158633 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.158819 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:07:49.162818 12508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:49.169224 12508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:49.190097 13014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.181368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:44947 (local address 127.0.0.1:59934)
0925 15:07:49.181917 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.181939 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.182074 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.183178 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.183185 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.183209 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.183629 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.183642 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.185225 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.185230 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.187447 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.187459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.189182 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.189188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.189377 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.189439 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.189553 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.189742 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":276,"threads_started":1}
I20260925 15:07:49.190418 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.181653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59934 (local address 127.4.205.195:44947)
0925 15:07:49.182625 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.182635 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.182664 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:07:49.182760 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.182770 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.182990 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.183810 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.184006 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.185031 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.187569 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.188971 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.189109 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.189300 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.189622 (+ 322us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.189784 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":664,"threads_started":1}
I20260925 15:07:49.191105 13012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.179994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.0.0.1:60758)
0925 15:07:49.180906 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.180944 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.181081 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.182060 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.182067 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.182112 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.182529 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.182541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.184800 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.184805 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.187339 (+ 2534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.187351 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.190358 (+ 3007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.190363 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.190510 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.190544 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.190671 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.190782 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"mutex_wait_us":23,"thread_start_us":519,"threads_started":1}
I20260925 15:07:49.191175 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.180243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.4.205.193:38829)
0925 15:07:49.180910 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.180918 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.181039 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:07:49.181284 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.181311 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.181625 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.181820 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.182761 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.184634 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.187480 (+ 2846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.188969 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.189109 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.189300 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.190750 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.190902 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":385,"threads_started":1}
I20260925 15:07:49.192312 13017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.182491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37873 (local address 127.0.0.1:49920)
0925 15:07:49.183004 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.183023 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.183128 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.185740 (+ 2612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.185744 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.185765 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.186227 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.186238 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.187842 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.187847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.189201 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.189211 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.191587 (+ 2376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.191593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.191766 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.191800 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.191894 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.191995 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":2,"thread_start_us":257,"threads_started":1}
I20260925 15:07:49.193008 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.182187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49920 (local address 127.4.205.194:37873)
0925 15:07:49.185198 (+ 3011us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.185205 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.185228 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:49.185311 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.185320 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.185531 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.185734 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.186408 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.187740 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.189364 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.191061 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.191216 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.191445 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.192602 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.192740 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2854,"thread_start_us":2734,"threads_started":1}
I20260925 15:07:49.194247 12923 tablet_service.cc:1467] Processing CreateTablet for tablet 0632b96b49974245951e498b293e148f (DEFAULT_TABLE table=test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.194738 12645 tablet_service.cc:1467] Processing CreateTablet for tablet 0632b96b49974245951e498b293e148f (DEFAULT_TABLE table=test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.195463 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0632b96b49974245951e498b293e148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.195865 12645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0632b96b49974245951e498b293e148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.199906 12784 tablet_service.cc:1467] Processing CreateTablet for tablet 0632b96b49974245951e498b293e148f (DEFAULT_TABLE table=test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.201093 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0632b96b49974245951e498b293e148f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.207022 13018 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Bootstrap starting.
I20260925 15:07:49.208314 13019 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Bootstrap starting.
I20260925 15:07:49.210639 13018 tablet_bootstrap.cc:654] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.211604 13019 tablet_bootstrap.cc:654] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.211850 13018 log.cc:824] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:49.212756 13019 log.cc:824] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:49.212975 13020 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Bootstrap starting.
I20260925 15:07:49.215813 13020 tablet_bootstrap.cc:654] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.216691 13020 log.cc:824] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:49.219046 13020 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: No bootstrap required, opened a new log
I20260925 15:07:49.219383 13020 ts_tablet_manager.cc:1397] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:07:49.222893 13019 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: No bootstrap required, opened a new log
I20260925 15:07:49.223160 13019 ts_tablet_manager.cc:1397] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260925 15:07:49.223928 13018 tablet_bootstrap.cc:492] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: No bootstrap required, opened a new log
I20260925 15:07:49.224197 13018 ts_tablet_manager.cc:1397] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.004s
I20260925 15:07:49.227293 13020 raft_consensus.cc:348] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.227694 13020 raft_consensus.cc:374] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.227809 13020 raft_consensus.cc:729] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.228493 13020 consensus_queue.cc:260] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.229967 13020 ts_tablet_manager.cc:1428] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Time spent starting tablet: real 0.010s user 0.011s sys 0.001s
I20260925 15:07:49.233160 13019 raft_consensus.cc:348] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.233534 13019 raft_consensus.cc:374] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.233701 13019 raft_consensus.cc:729] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87541fdc92a478fb1e45b35f01baf8d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.234328 13019 consensus_queue.cc:260] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.236155 13019 ts_tablet_manager.cc:1428] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Time spent starting tablet: real 0.013s user 0.010s sys 0.003s
I20260925 15:07:49.236665 13018 raft_consensus.cc:348] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.237021 13018 raft_consensus.cc:374] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.237126 13018 raft_consensus.cc:729] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.237715 13018 consensus_queue.cc:260] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.239300 13018 ts_tablet_manager.cc:1428] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260925 15:07:49.303411 12855 tablet.cc:2369] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:49.319836 12994 tablet.cc:2369] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:07:49.321866 12716 tablet.cc:2369] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:07:49.419533 13026 raft_consensus.cc:482] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:49.419871 13026 raft_consensus.cc:504] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.421640 13026 leader_election.cc:288] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947), ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873)
I20260925 15:07:49.427848 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.422024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:48749 (local address 127.4.205.195:44947)
0925 15:07:49.422206 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.422213 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.422579 (+ 366us) server_negotiation.cc:407] Connection header received
0925 15:07:49.422909 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.422917 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.423007 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.423071 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.423731 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.424406 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.426267 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.426996 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.427172 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.427240 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.427499 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.427618 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:07:49.427987 13027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.421849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:44947 (local address 127.4.205.193:48749)
0925 15:07:49.422477 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.422499 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.422697 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.423256 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.423259 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.423277 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.423593 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.423600 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.424531 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.424539 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.426117 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.426128 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.427187 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.427193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.427347 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.427370 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.427479 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.427592 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"spinlock_wait_cycles":17792,"thread_start_us":308,"threads_started":1}
I20260925 15:07:49.428463 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.422719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:45807 (local address 127.4.205.194:37873)
0925 15:07:49.422961 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.422967 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.423283 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:07:49.423490 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.423496 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.423564 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.423653 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.424125 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.424920 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.426810 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.427513 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.427681 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.427750 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.428086 (+ 336us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.428222 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:07:49.429092 13028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.422596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37873 (local address 127.4.205.193:45807)
0925 15:07:49.423186 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.423199 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.423346 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.423664 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.423668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.423689 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.423988 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.423995 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.425043 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.425049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.426671 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.426681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.427723 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.427730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.427887 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.427942 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.428628 (+ 686us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.428754 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":326,"threads_started":1}
I20260925 15:07:49.429045 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0632b96b49974245951e498b293e148f" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d" is_pre_election: true
I20260925 15:07:49.429618 12948 raft_consensus.cc:2454] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:49.430099 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0632b96b49974245951e498b293e148f" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c" is_pre_election: true
I20260925 15:07:49.430223 12597 leader_election.cc:302] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:49.430562 12804 raft_consensus.cc:2454] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:49.430588 13026 raft_consensus.cc:2781] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:49.430720 13026 raft_consensus.cc:482] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:49.430836 13026 raft_consensus.cc:3037] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.433212 13026 raft_consensus.cc:504] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.433782 13026 leader_election.cc:288] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Requested vote from peers b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947), ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873)
I20260925 15:07:49.434137 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0632b96b49974245951e498b293e148f" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d"
I20260925 15:07:49.434227 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0632b96b49974245951e498b293e148f" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c"
I20260925 15:07:49.434309 12948 raft_consensus.cc:3037] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.434445 12804 raft_consensus.cc:3037] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.436378 12948 raft_consensus.cc:2454] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:49.436625 12804 raft_consensus.cc:2454] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:49.436740 12597 leader_election.cc:302] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:49.437036 13026 raft_consensus.cc:2781] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:49.437456 13026 raft_consensus.cc:686] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Running, Role: LEADER
I20260925 15:07:49.437919 13026 consensus_queue.cc:237] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } }
I20260925 15:07:49.442360 12508 catalog_manager.cc:5495] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:49.490440 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.485371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60764 (local address 127.4.205.193:38829)
0925 15:07:49.485580 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.485586 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.485701 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:07:49.485912 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.485916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.486013 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.486129 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:49.486414 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.487126 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.488426 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.488862 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.488959 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.489311 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.489879 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.489994 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.490099 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.490156 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:07:49.497516 12804 raft_consensus.cc:1261] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.497721 12948 raft_consensus.cc:1261] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.498515 13029 consensus_queue.cc:1035] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.498934 13026 consensus_queue.cc:1035] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.507555 13032 mvcc.cc:205] Tried to move back new op lower bound from 7333268969452961792 to 7333268969220726784. Current Snapshot: MvccSnapshot[applied={T|T < 7333268969452961792}]
I20260925 15:07:49.522800 12508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:49.523047 12508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:49.526463 12508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Sending DeleteTablet for 3 replicas of tablet 0632b96b49974245951e498b293e148f
I20260925 15:07:49.527482 12923 tablet_service.cc:1514] Processing DeleteTablet for tablet 0632b96b49974245951e498b293e148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260925 15:07:49.527597 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0632b96b49974245951e498b293e148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49920
I20260925 15:07:49.527617 12645 tablet_service.cc:1514] Processing DeleteTablet for tablet 0632b96b49974245951e498b293e148f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60758
I20260925 15:07:49.529110 12508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:49.529951 12508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:49.539698 12783 tablet_service.cc:1467] Processing CreateTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf (DEFAULT_TABLE table=test-table [id=10202b94d13d4cbbb540839a3d082f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.540026 12922 tablet_service.cc:1467] Processing CreateTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf (DEFAULT_TABLE table=test-table [id=10202b94d13d4cbbb540839a3d082f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.540241 12783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127c72a8a3ae4efe92edd108233d0fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.540557 12922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127c72a8a3ae4efe92edd108233d0fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.541806 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf (DEFAULT_TABLE table=test-table [id=10202b94d13d4cbbb540839a3d082f8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:49.542378 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127c72a8a3ae4efe92edd108233d0fcf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.546290 13043 tablet_replica.cc:330] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: stopping tablet replica
I20260925 15:07:49.547078 13019 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Bootstrap starting.
I20260925 15:07:49.547194 13020 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Bootstrap starting.
I20260925 15:07:49.548653 13019 tablet_bootstrap.cc:654] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.549170 13020 tablet_bootstrap.cc:654] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.546813 13043 raft_consensus.cc:2229] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:49.555378 13043 raft_consensus.cc:2258] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.559640 13041 tablet_replica.cc:330] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: stopping tablet replica
I20260925 15:07:49.560053 13041 raft_consensus.cc:2229] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:49.560493 13041 raft_consensus.cc:2258] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.561477 13043 ts_tablet_manager.cc:1905] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.563470 13018 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Bootstrap starting.
I20260925 15:07:49.565816 13019 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: No bootstrap required, opened a new log
I20260925 15:07:49.565953 13019 ts_tablet_manager.cc:1397] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.002s
I20260925 15:07:49.566975 13019 raft_consensus.cc:348] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.567199 13019 raft_consensus.cc:374] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.567209 13018 tablet_bootstrap.cc:654] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.567296 13019 raft_consensus.cc:729] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87541fdc92a478fb1e45b35f01baf8d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.567564 13019 consensus_queue.cc:260] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.569918 13020 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: No bootstrap required, opened a new log
I20260925 15:07:49.570099 13020 ts_tablet_manager.cc:1397] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.011s
I20260925 15:07:49.570381 13019 ts_tablet_manager.cc:1428] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260925 15:07:49.571017 13020 raft_consensus.cc:348] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.571259 13020 raft_consensus.cc:374] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.571367 13020 raft_consensus.cc:729] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.571653 13020 consensus_queue.cc:260] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.572364 13020 ts_tablet_manager.cc:1428] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:07:49.575387 13042 tablet_replica.cc:330] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: stopping tablet replica
I20260925 15:07:49.575794 13042 raft_consensus.cc:2229] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:49.576452 13042 raft_consensus.cc:2258] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.577921 13041 ts_tablet_manager.cc:1905] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.578436 13043 ts_tablet_manager.cc:1918] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:49.578563 13042 ts_tablet_manager.cc:1905] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.578589 13043 log.cc:1196] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/0632b96b49974245951e498b293e148f
I20260925 15:07:49.579164 13043 ts_tablet_manager.cc:1939] T 0632b96b49974245951e498b293e148f P ae7b58600bbe4cf38c12cd53518ba26d: Deleting consensus metadata
I20260925 15:07:49.585893 13042 ts_tablet_manager.cc:1918] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:49.586035 12494 catalog_manager.cc:4841] TS ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829): tablet 0632b96b49974245951e498b293e148f (table test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]) successfully deleted
I20260925 15:07:49.586196 13042 log.cc:1196] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/0632b96b49974245951e498b293e148f
I20260925 15:07:49.586745 13042 ts_tablet_manager.cc:1939] T 0632b96b49974245951e498b293e148f P ad0319b5273145b8ba8cf56f74086d1c: Deleting consensus metadata
I20260925 15:07:49.587610 13018 tablet_bootstrap.cc:492] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: No bootstrap required, opened a new log
I20260925 15:07:49.587791 13018 ts_tablet_manager.cc:1397] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.008s
I20260925 15:07:49.588222 12494 catalog_manager.cc:4841] TS ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873): tablet 0632b96b49974245951e498b293e148f (table test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]) successfully deleted
I20260925 15:07:49.588714 13041 ts_tablet_manager.cc:1918] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:49.588871 13041 log.cc:1196] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/0632b96b49974245951e498b293e148f
I20260925 15:07:49.588759 13018 raft_consensus.cc:348] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.589093 13018 raft_consensus.cc:374] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.589229 13018 raft_consensus.cc:729] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.589363 13041 ts_tablet_manager.cc:1939] T 0632b96b49974245951e498b293e148f P b87541fdc92a478fb1e45b35f01baf8d: Deleting consensus metadata
I20260925 15:07:49.589525 13018 consensus_queue.cc:260] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.590512 13024 raft_consensus.cc:482] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:49.590654 13024 raft_consensus.cc:504] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.590188 13018 ts_tablet_manager.cc:1428] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:49.591900 12494 catalog_manager.cc:4841] TS b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947): tablet 0632b96b49974245951e498b293e148f (table test-table [id=b7593a1c1e554e5b9189aad3b719ec5d]) successfully deleted
I20260925 15:07:49.592214 13024 leader_election.cc:288] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:49.599529 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.592977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:54143 (local address 127.4.205.193:38829)
0925 15:07:49.593130 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.593137 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.593156 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:07:49.593381 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.593386 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.593455 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.593518 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.594405 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.595363 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.597520 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.598448 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.598624 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.598688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.599098 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.599242 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:07:49.600152 13025 raft_consensus.cc:482] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:49.600299 13047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.592659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.4.205.194:54143)
0925 15:07:49.593078 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.593097 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.593217 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.593759 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.593763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.593785 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.594251 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.594263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.595501 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.595506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.597389 (+ 1883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.597399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.598643 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.598648 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.598826 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.598883 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.599744 (+ 861us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.599888 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":20,"thread_start_us":502,"threads_started":1}
I20260925 15:07:49.600407 13025 raft_consensus.cc:504] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.600989 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "ad0319b5273145b8ba8cf56f74086d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" is_pre_election: true
I20260925 15:07:49.601337 13048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.593463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:44947 (local address 127.4.205.194:35125)
0925 15:07:49.594386 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.594404 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.594510 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.594831 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.594835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.594859 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.595185 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.595196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.596606 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.596611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.598711 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.598725 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.600557 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.600566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.600726 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.600754 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.600847 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.600956 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":787,"mutex_wait_us":1,"thread_start_us":611,"threads_started":1}
I20260925 15:07:49.601408 12670 raft_consensus.cc:2454] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad0319b5273145b8ba8cf56f74086d1c in term 0.
I20260925 15:07:49.601750 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.593774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:35125 (local address 127.4.205.195:44947)
0925 15:07:49.593930 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.593936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.594471 (+ 535us) server_negotiation.cc:407] Connection header received
0925 15:07:49.594622 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.594626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.594694 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.594792 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.595363 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.596470 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.598870 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.600111 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.600293 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.600381 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.600894 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.601022 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:07:49.602010 12737 leader_election.cc:302] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad0319b5273145b8ba8cf56f74086d1c, ae7b58600bbe4cf38c12cd53518ba26d; no voters:
I20260925 15:07:49.602080 13025 leader_election.cc:288] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829), ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873)
I20260925 15:07:49.602355 13024 raft_consensus.cc:2781] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:49.602309 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "ad0319b5273145b8ba8cf56f74086d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d" is_pre_election: true
I20260925 15:07:49.602458 13024 raft_consensus.cc:482] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:49.602545 13024 raft_consensus.cc:3037] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.602550 12948 raft_consensus.cc:2454] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ad0319b5273145b8ba8cf56f74086d1c in term 0.
I20260925 15:07:49.605409 13024 raft_consensus.cc:504] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.606218 13024 leader_election.cc:288] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [CANDIDATE]: Term 1 election: Requested vote from peers ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:49.606828 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "ad0319b5273145b8ba8cf56f74086d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b58600bbe4cf38c12cd53518ba26d"
I20260925 15:07:49.606913 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "ad0319b5273145b8ba8cf56f74086d1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d"
I20260925 15:07:49.607010 12670 raft_consensus.cc:3037] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.607079 12948 raft_consensus.cc:3037] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.609947 12948 raft_consensus.cc:2454] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0319b5273145b8ba8cf56f74086d1c in term 1.
I20260925 15:07:49.609997 12670 raft_consensus.cc:2454] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ad0319b5273145b8ba8cf56f74086d1c in term 1.
I20260925 15:07:49.610379 12737 leader_election.cc:302] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad0319b5273145b8ba8cf56f74086d1c, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:49.610646 13024 raft_consensus.cc:2781] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:49.611070 13024 raft_consensus.cc:686] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 1 LEADER]: Becoming Leader. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Running, Role: LEADER
I20260925 15:07:49.611402 13024 consensus_queue.cc:237] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.614511 12507 catalog_manager.cc:5495] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c reported cstate change: term changed from 0 to 1, leader changed from <none> to ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194). New cstate: current_term: 1 leader_uuid: "ad0319b5273145b8ba8cf56f74086d1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:49.621276 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.602917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:48795 (local address 127.4.205.193:38829)
0925 15:07:49.603104 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.603109 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.613853 (+ 10744us) server_negotiation.cc:407] Connection header received
0925 15:07:49.614085 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.614089 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.614152 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.614216 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.615008 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.615938 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.618819 (+ 2881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.620132 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.620249 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.620313 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.620822 (+ 509us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.620947 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:07:49.621944 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.602523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.4.205.195:48795)
0925 15:07:49.606667 (+ 4144us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.613784 (+ 7117us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.613947 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.614429 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.614434 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.614457 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.614814 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.614825 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.616669 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.616673 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.618626 (+ 1953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.618636 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.620499 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.620506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.620667 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.620696 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.621458 (+ 762us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.621616 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4040,"thread_start_us":3923,"threads_started":1}
I20260925 15:07:49.630374 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.610248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:32901 (local address 127.4.205.194:37873)
0925 15:07:49.610428 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.610433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.622252 (+ 11819us) server_negotiation.cc:407] Connection header received
0925 15:07:49.622487 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.622492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.622569 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.622632 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:07:49.623154 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.624592 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.627434 (+ 2842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.628909 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.629070 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.629167 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.629848 (+ 681us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.630020 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:07:49.631101 13049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.606881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37873 (local address 127.4.205.195:32901)
0925 15:07:49.622165 (+ 15284us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.622184 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.622328 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.622810 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.622813 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.622836 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:07:49.622985 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.622996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.624832 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.624839 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.627239 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.627257 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.629324 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.629345 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.629632 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.629683 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.630569 (+ 886us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.630741 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15192,"threads_started":1}
I20260925 15:07:49.631898 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "b87541fdc92a478fb1e45b35f01baf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c" is_pre_election: true
I20260925 15:07:49.632694 12670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127c72a8a3ae4efe92edd108233d0fcf" candidate_uuid: "b87541fdc92a478fb1e45b35f01baf8d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" is_pre_election: true
I20260925 15:07:49.633016 12670 raft_consensus.cc:2379] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b87541fdc92a478fb1e45b35f01baf8d in current term 1: Already voted for candidate ad0319b5273145b8ba8cf56f74086d1c in this term.
I20260925 15:07:49.633715 12876 leader_election.cc:302] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b87541fdc92a478fb1e45b35f01baf8d; no voters: ad0319b5273145b8ba8cf56f74086d1c, ae7b58600bbe4cf38c12cd53518ba26d
I20260925 15:07:49.634104 13025 raft_consensus.cc:2726] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
Sep 25 15:07:49 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348866, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.254@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:49.664108 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.651793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35929 (local address 127.0.0.1:43684)
0925 15:07:49.652366 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.652380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.652519 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.652794 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.652798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.653405 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:49.653699 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.653708 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.654588 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.654595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.655418 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.655428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.656076 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.656081 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.656217 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.656859 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:49.656875 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:49.659019 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:49.661761 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:49.661774 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:49.661782 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:49.662174 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:49.662502 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:49.662509 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:49.662514 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:49.662667 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:49.663125 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:49.663138 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:49.663294 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.663464 (+ 170us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.663704 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":320,"threads_started":1}
I20260925 15:07:49.664115 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.651687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.4.205.254:35929)
0925 15:07:49.651820 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.651824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.652457 (+ 633us) server_negotiation.cc:407] Connection header received
0925 15:07:49.652633 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.652636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.652681 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.652773 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:49.653809 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.654463 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.655537 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.655922 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.656047 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.659187 (+ 3140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:49.659191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:49.659195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:49.659219 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:49.661658 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.662284 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.662292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.662296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.662404 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:49.662770 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:49.662775 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:49.662779 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:49.663011 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:49.663165 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.663537 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.663811 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":116,"server-negotiator.queue_time_us":47}
I20260925 15:07:49.668766 12507 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:43684:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 15:07:49.670045 12507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:49.676476 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 9164f139fb2f4053afd94776c0f72161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9d75dbc7f7c4d9088e82477d235d746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:49.676599 12784 tablet_service.cc:1467] Processing CreateTablet for tablet 9164f139fb2f4053afd94776c0f72161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9d75dbc7f7c4d9088e82477d235d746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:49.676756 12923 tablet_service.cc:1467] Processing CreateTablet for tablet 9164f139fb2f4053afd94776c0f72161 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f9d75dbc7f7c4d9088e82477d235d746]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 15:07:49.677033 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9164f139fb2f4053afd94776c0f72161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.677120 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9164f139fb2f4053afd94776c0f72161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.677206 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9164f139fb2f4053afd94776c0f72161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:49.684031 13020 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: Bootstrap starting.
I20260925 15:07:49.684048 13019 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: Bootstrap starting.
I20260925 15:07:49.685379 13018 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: Bootstrap starting.
I20260925 15:07:49.685642 13019 tablet_bootstrap.cc:654] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.685654 13020 tablet_bootstrap.cc:654] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.687322 13018 tablet_bootstrap.cc:654] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:49.699175 13020 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: No bootstrap required, opened a new log
I20260925 15:07:49.699359 13020 ts_tablet_manager.cc:1397] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.010s
I20260925 15:07:49.699783 13019 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: No bootstrap required, opened a new log
I20260925 15:07:49.699837 13018 tablet_bootstrap.cc:492] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: No bootstrap required, opened a new log
I20260925 15:07:49.699929 13019 ts_tablet_manager.cc:1397] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20260925 15:07:49.700001 13018 ts_tablet_manager.cc:1397] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.005s
I20260925 15:07:49.700604 13019 raft_consensus.cc:348] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.700834 13019 raft_consensus.cc:374] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.700716 13018 raft_consensus.cc:348] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.700896 13019 raft_consensus.cc:729] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87541fdc92a478fb1e45b35f01baf8d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.700954 13018 raft_consensus.cc:374] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.701025 13018 raft_consensus.cc:729] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.700927 13020 raft_consensus.cc:348] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.701165 13020 raft_consensus.cc:374] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:49.701233 13020 raft_consensus.cc:729] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Initialized, Role: FOLLOWER
I20260925 15:07:49.701159 13019 consensus_queue.cc:260] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.701278 13018 consensus_queue.cc:260] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.701706 13029 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.701954 13018 ts_tablet_manager.cc:1428] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:49.702034 13029 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.701910 13025 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.702143 13025 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.701517 13020 consensus_queue.cc:260] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.702980 13024 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.703224 13024 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.704125 13020 ts_tablet_manager.cc:1428] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 15:07:49.705088 13019 ts_tablet_manager.cc:1428] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260925 15:07:49.746624 13029 raft_consensus.cc:482] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:49.746865 13029 raft_consensus.cc:504] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.747498 13029 leader_election.cc:288] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:49.747973 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9164f139fb2f4053afd94776c0f72161" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d" is_pre_election: true
I20260925 15:07:49.748008 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9164f139fb2f4053afd94776c0f72161" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c" is_pre_election: true
I20260925 15:07:49.748199 12948 raft_consensus.cc:2454] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:49.748288 12804 raft_consensus.cc:2454] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:49.748585 12597 leader_election.cc:302] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:49.748870 13029 raft_consensus.cc:2781] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:49.748965 13029 raft_consensus.cc:482] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:49.749073 13029 raft_consensus.cc:3037] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.751168 13029 raft_consensus.cc:504] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.751729 13029 leader_election.cc:288] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Requested vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:49.752035 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9164f139fb2f4053afd94776c0f72161" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c"
I20260925 15:07:49.752206 12804 raft_consensus.cc:3037] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.752166 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9164f139fb2f4053afd94776c0f72161" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d"
I20260925 15:07:49.752378 12948 raft_consensus.cc:3037] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:49.754477 12804 raft_consensus.cc:2454] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:49.755242 12948 raft_consensus.cc:2454] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:49.755564 12597 leader_election.cc:302] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:49.755849 13029 raft_consensus.cc:2781] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:49.756040 13029 raft_consensus.cc:686] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Running, Role: LEADER
I20260925 15:07:49.756349 13029 consensus_queue.cc:237] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:49.757977 13026 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: TxnStatusTablet state changed. Reason: New leader ae7b58600bbe4cf38c12cd53518ba26d. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.758288 13026 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:49.759261 12506 catalog_manager.cc:5495] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:49.765961 13063 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 15:07:49.766196 13063 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 15:07:49.780857 13058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.775697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.0.0.1:60774)
0925 15:07:49.775915 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.775934 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.776053 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.776394 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.776400 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.776413 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:49.776603 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.776612 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.777716 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.777723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.778861 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.778872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.779672 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.779677 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.779881 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.779933 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.780313 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.780332 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.780438 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.780572 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:07:49.780849 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.775797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60774 (local address 127.4.205.193:38829)
0925 15:07:49.775993 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.776000 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.776015 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:07:49.776205 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.776208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.776260 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.776348 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:49.776719 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.777529 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.779012 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.779529 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.779672 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.780116 (+ 444us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.780207 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.780308 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.780458 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.780531 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 15:07:49.783912 12804 raft_consensus.cc:1261] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.784119 12948 raft_consensus.cc:1261] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.784530 13026 consensus_queue.cc:1035] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.784803 13029 consensus_queue.cc:1035] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.790347 13024 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: TxnStatusTablet state changed. Reason: New leader ae7b58600bbe4cf38c12cd53518ba26d. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.790637 13024 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.794174 13034 mvcc.cc:205] Tried to move back new op lower bound from 7333268970629791744 to 7333268970524270592. Current Snapshot: MvccSnapshot[applied={T|T < 7333268970629791744 or (T in {7333268970629791744})}]
I20260925 15:07:49.794826 13024 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.795063 13024 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ad0319b5273145b8ba8cf56f74086d1c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.795913 13026 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.796155 13026 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:49.801999 13025 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: TxnStatusTablet state changed. Reason: New leader ae7b58600bbe4cf38c12cd53518ba26d. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.802335 13025 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.802775 13033 mvcc.cc:205] Tried to move back new op lower bound from 7333268970629791744 to 7333268970524270592. Current Snapshot: MvccSnapshot[applied={T|T < 7333268970629791744}]
I20260925 15:07:49.802515 13026 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.803558 13025 tablet_replica.cc:439] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } } }
I20260925 15:07:49.803748 13025 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P b87541fdc92a478fb1e45b35f01baf8d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 15:07:49.803882 13026 tablet_replica.cc:442] T 9164f139fb2f4053afd94776c0f72161 P ae7b58600bbe4cf38c12cd53518ba26d: This TxnStatusTablet replica's current role is: LEADER
I20260925 15:07:49.808205 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.798987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49934 (local address 127.4.205.194:37873)
0925 15:07:49.799129 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.799135 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.799374 (+ 239us) server_negotiation.cc:407] Connection header received
0925 15:07:49.799614 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.799620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.799740 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.799886 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:49.800311 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.801338 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.804458 (+ 3120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.805337 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.805472 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.806785 (+ 1313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.807598 (+ 813us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.807754 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.807879 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.807950 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:07:49.823163 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.817649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:34291 (local address 127.4.205.193:38829)
0925 15:07:49.817816 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.817821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.818319 (+ 498us) server_negotiation.cc:407] Connection header received
0925 15:07:49.818423 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.818430 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.818507 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.818599 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:49.819025 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.820197 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.821442 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.821940 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.822090 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.822463 (+ 373us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.822566 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.822674 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.822798 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.822874 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:07:49.823249 13070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.817612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.4.205.194:34291)
0925 15:07:49.818220 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.818235 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.818330 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.818611 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.818614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.818622 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:49.818882 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.818888 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.820304 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.820307 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.821294 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.821304 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.822115 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.822121 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.822245 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.822290 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.822670 (+ 380us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.822682 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.822776 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.822892 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":339,"threads_started":1}
I20260925 15:07:49.831830 12948 raft_consensus.cc:1261] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Refusing update from remote peer ad0319b5273145b8ba8cf56f74086d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.832024 12670 raft_consensus.cc:1261] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Refusing update from remote peer ad0319b5273145b8ba8cf56f74086d1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:49.832391 13051 consensus_queue.cc:1035] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.832998 13051 consensus_queue.cc:1035] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:49.877278 13016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.870798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:39315 (local address 127.4.205.194:37873)
0925 15:07:49.870987 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.870995 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.871676 (+ 681us) server_negotiation.cc:407] Connection header received
0925 15:07:49.871814 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.871819 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.871895 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.871992 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:49.872432 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.873499 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.875525 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.876045 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.876174 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.876586 (+ 412us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.876686 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.876773 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:49.876895 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:07:49.876965 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:07:49.877454 13074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:49.870875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37873 (local address 127.4.205.193:39315)
0925 15:07:49.871570 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:49.871590 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:49.871703 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:49.871997 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:49.872000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:49.872008 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:49.872297 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.872303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.873646 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.873653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.875391 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:49.875405 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.876222 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.876233 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:49.876381 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.876427 (+ 46us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:49.876775 (+ 348us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:49.876787 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:07:49.876877 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:07:49.877016 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":359,"threads_started":1}
I20260925 15:07:49.927145 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:49.927377 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:49.930508 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Sending DeleteTablet for 3 replicas of tablet 127c72a8a3ae4efe92edd108233d0fcf
I20260925 15:07:49.931259 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49920
I20260925 15:07:49.931421 12923 tablet_service.cc:1514] Processing DeleteTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260925 15:07:49.931555 13042 tablet_replica.cc:330] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: stopping tablet replica
I20260925 15:07:49.931681 13041 tablet_replica.cc:330] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: stopping tablet replica
I20260925 15:07:49.931751 13042 raft_consensus.cc:2229] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:49.931897 13041 raft_consensus.cc:2229] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:49.932078 13041 raft_consensus.cc:2258] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.932261 13042 raft_consensus.cc:2258] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.932282 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet 127c72a8a3ae4efe92edd108233d0fcf with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:49 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60758
I20260925 15:07:49.932538 13043 tablet_replica.cc:330] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: stopping tablet replica
I20260925 15:07:49.932744 13043 raft_consensus.cc:2229] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:49.932940 13043 raft_consensus.cc:2258] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:49.933869 13041 ts_tablet_manager.cc:1905] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.934026 13043 ts_tablet_manager.cc:1905] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.939827 13042 ts_tablet_manager.cc:1905] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:49.940675 13043 ts_tablet_manager.cc:1918] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:49.940835 13043 log.cc:1196] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/127c72a8a3ae4efe92edd108233d0fcf
I20260925 15:07:49.941226 13043 ts_tablet_manager.cc:1939] T 127c72a8a3ae4efe92edd108233d0fcf P ae7b58600bbe4cf38c12cd53518ba26d: Deleting consensus metadata
I20260925 15:07:49.942361 12494 catalog_manager.cc:4841] TS ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829): tablet 127c72a8a3ae4efe92edd108233d0fcf (table test-table [id=10202b94d13d4cbbb540839a3d082f8b]) successfully deleted
I20260925 15:07:49.948503 13042 ts_tablet_manager.cc:1918] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:49.948647 13042 log.cc:1196] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/127c72a8a3ae4efe92edd108233d0fcf
I20260925 15:07:49.949080 13042 ts_tablet_manager.cc:1939] T 127c72a8a3ae4efe92edd108233d0fcf P ad0319b5273145b8ba8cf56f74086d1c: Deleting consensus metadata
I20260925 15:07:49.951323 13041 ts_tablet_manager.cc:1918] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:49.951467 13041 log.cc:1196] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/127c72a8a3ae4efe92edd108233d0fcf
I20260925 15:07:49.951930 13041 ts_tablet_manager.cc:1939] T 127c72a8a3ae4efe92edd108233d0fcf P b87541fdc92a478fb1e45b35f01baf8d: Deleting consensus metadata
I20260925 15:07:49.953076 12494 catalog_manager.cc:4841] TS b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947): tablet 127c72a8a3ae4efe92edd108233d0fcf (table test-table [id=10202b94d13d4cbbb540839a3d082f8b]) successfully deleted
I20260925 15:07:49.953987 12494 catalog_manager.cc:4841] TS ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873): tablet 127c72a8a3ae4efe92edd108233d0fcf (table test-table [id=10202b94d13d4cbbb540839a3d082f8b]) successfully deleted
Sep 25 15:07:49 dist-test-slave-64fx krb5kdc[12463](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 15:07:49 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:49 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:49.999444 12603 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:49.984998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43692 (local address 127.4.205.254:35929)
0925 15:07:49.985194 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:49.985203 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:49.985718 (+ 515us) server_negotiation.cc:407] Connection header received
0925 15:07:49.985913 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:49.985919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:49.985994 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:49.986095 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:49.987534 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.988465 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.989780 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:49.990210 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:49.990352 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:49.999195 (+ 8843us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43692: BlockingRecv error: recv got EOF from 127.0.0.1:43692 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:07:50 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM
I20260925 15:07:50.015892 12603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.003029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43708 (local address 127.4.205.254:35929)
0925 15:07:50.003195 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.003200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.003217 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:50.003391 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.003397 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.003468 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.003519 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.004660 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.005427 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.006658 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.007051 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.007168 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.010687 (+ 3519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.010692 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.010695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.010727 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.013280 (+ 2553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.013997 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.014007 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.014009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.014114 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.014465 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.014470 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.014476 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.014725 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.015002 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.015295 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.015510 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:07:50.019157 12507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:07:50.019742 12507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:50.026634 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 91b2beceae764a3390c91669580ad643 (DEFAULT_TABLE table=test-table [id=0334fb751d434a23b1f265df1035f03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.027098 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b2beceae764a3390c91669580ad643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.027316 12784 tablet_service.cc:1467] Processing CreateTablet for tablet 91b2beceae764a3390c91669580ad643 (DEFAULT_TABLE table=test-table [id=0334fb751d434a23b1f265df1035f03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.027704 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b2beceae764a3390c91669580ad643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.029935 12923 tablet_service.cc:1467] Processing CreateTablet for tablet 91b2beceae764a3390c91669580ad643 (DEFAULT_TABLE table=test-table [id=0334fb751d434a23b1f265df1035f03b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.030321 12923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91b2beceae764a3390c91669580ad643. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.033691 13018 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Bootstrap starting.
I20260925 15:07:50.035326 13018 tablet_bootstrap.cc:654] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.046972 13018 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: No bootstrap required, opened a new log
I20260925 15:07:50.047125 13018 ts_tablet_manager.cc:1397] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260925 15:07:50.047766 13020 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Bootstrap starting.
I20260925 15:07:50.047895 13018 raft_consensus.cc:348] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.048174 13018 raft_consensus.cc:374] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.048265 13018 raft_consensus.cc:729] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.048472 13018 consensus_queue.cc:260] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.049021 13018 ts_tablet_manager.cc:1428] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:07:50.049422 13020 tablet_bootstrap.cc:654] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.052762 13019 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Bootstrap starting.
I20260925 15:07:50.054209 13020 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: No bootstrap required, opened a new log
I20260925 15:07:50.054231 13019 tablet_bootstrap.cc:654] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.054350 13020 ts_tablet_manager.cc:1397] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.001s
I20260925 15:07:50.055176 13020 raft_consensus.cc:348] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.055508 13020 raft_consensus.cc:374] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.055624 13020 raft_consensus.cc:729] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.056528 13020 consensus_queue.cc:260] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.058157 13020 ts_tablet_manager.cc:1428] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260925 15:07:50.059522 13019 tablet_bootstrap.cc:492] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: No bootstrap required, opened a new log
I20260925 15:07:50.059650 13019 ts_tablet_manager.cc:1397] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 15:07:50.060364 13019 raft_consensus.cc:348] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.060603 13019 raft_consensus.cc:374] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.060691 13019 raft_consensus.cc:729] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87541fdc92a478fb1e45b35f01baf8d, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.060928 13019 consensus_queue.cc:260] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.063230 13019 ts_tablet_manager.cc:1428] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:50.141933 13029 raft_consensus.cc:482] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:50.142187 13029 raft_consensus.cc:504] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.142969 13029 leader_election.cc:288] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:50.143567 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b2beceae764a3390c91669580ad643" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c" is_pre_election: true
I20260925 15:07:50.143584 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b2beceae764a3390c91669580ad643" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d" is_pre_election: true
I20260925 15:07:50.143800 12804 raft_consensus.cc:2454] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:50.143832 12948 raft_consensus.cc:2454] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:50.144286 12597 leader_election.cc:302] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:50.144773 13029 raft_consensus.cc:2781] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:50.144883 13029 raft_consensus.cc:482] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:50.144979 13029 raft_consensus.cc:3037] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.148082 13029 raft_consensus.cc:504] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.148734 13029 leader_election.cc:288] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Requested vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:50.149201 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b2beceae764a3390c91669580ad643" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d"
I20260925 15:07:50.149288 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91b2beceae764a3390c91669580ad643" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c"
I20260925 15:07:50.149407 12948 raft_consensus.cc:3037] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.149518 12804 raft_consensus.cc:3037] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.152237 12948 raft_consensus.cc:2454] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:50.152630 12804 raft_consensus.cc:2454] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:50.152608 12597 leader_election.cc:302] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:50.153018 13026 raft_consensus.cc:2781] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:50.153177 13026 raft_consensus.cc:686] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Running, Role: LEADER
I20260925 15:07:50.153430 13026 consensus_queue.cc:237] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.156283 12508 catalog_manager.cc:5495] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:50.181408 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.174852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60776 (local address 127.4.205.193:38829)
0925 15:07:50.175038 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.175044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.175300 (+ 256us) server_negotiation.cc:407] Connection header received
0925 15:07:50.175509 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.175515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.175604 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.175701 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:50.176093 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.177298 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.179245 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.179946 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.180090 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.180579 (+ 489us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:50.180728 (+ 149us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:50.180833 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.180980 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.181058 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:07:50.184392 12948 raft_consensus.cc:1261] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:50.184984 12804 raft_consensus.cc:1261] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:50.185022 13029 consensus_queue.cc:1035] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:50.185680 13029 consensus_queue.cc:1035] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:50.206936 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:50.207141 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:50.211480 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Sending DeleteTablet for 3 replicas of tablet 91b2beceae764a3390c91669580ad643
I20260925 15:07:50.212365 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b2beceae764a3390c91669580ad643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49920
I20260925 15:07:50.212414 12923 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b2beceae764a3390c91669580ad643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260925 15:07:50.212579 12922 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b2beceae764a3390c91669580ad643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260925 15:07:50.212639 13042 tablet_replica.cc:330] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: stopping tablet replica
I20260925 15:07:50.212810 13042 raft_consensus.cc:2229] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:50.213016 13042 raft_consensus.cc:2258] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.213382 13041 tablet_replica.cc:330] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: stopping tablet replica
I20260925 15:07:50.213526 13041 raft_consensus.cc:2229] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:50.213745 13041 raft_consensus.cc:2258] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.214020 12644 tablet_service.cc:1514] Processing DeleteTablet for tablet 91b2beceae764a3390c91669580ad643 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60758
I20260925 15:07:50.214432 13043 tablet_replica.cc:330] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: stopping tablet replica
I20260925 15:07:50.214612 13043 raft_consensus.cc:2229] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:50.214934 13043 raft_consensus.cc:2258] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.216499 12508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 15:07:50.216898 13042 ts_tablet_manager.cc:1905] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:07:50.217090 12508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:07:50.223897 12784 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 (DEFAULT_TABLE table=test-table [id=0fd4af541b9942caa720b8386ca4d29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.223984 12922 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 (DEFAULT_TABLE table=test-table [id=0fd4af541b9942caa720b8386ca4d29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.224303 12784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5f368f2aa440618513ec384eaba9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.224395 12922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5f368f2aa440618513ec384eaba9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.229995 12644 tablet_service.cc:1467] Processing CreateTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 (DEFAULT_TABLE table=test-table [id=0fd4af541b9942caa720b8386ca4d29c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:07:50.230379 12644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f5f368f2aa440618513ec384eaba9a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:07:50.232283 13043 ts_tablet_manager.cc:1905] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:50.233737 13019 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Bootstrap starting.
I20260925 15:07:50.235246 13019 tablet_bootstrap.cc:654] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.239135 13043 ts_tablet_manager.cc:1918] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:50.239260 13043 log.cc:1196] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/91b2beceae764a3390c91669580ad643
I20260925 15:07:50.239722 13043 ts_tablet_manager.cc:1939] T 91b2beceae764a3390c91669580ad643 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting consensus metadata
I20260925 15:07:50.241444 12494 catalog_manager.cc:4841] TS ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829): tablet 91b2beceae764a3390c91669580ad643 (table test-table [id=0334fb751d434a23b1f265df1035f03b]) successfully deleted
I20260925 15:07:50.244385 13018 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Bootstrap starting.
I20260925 15:07:50.245501 13020 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Bootstrap starting.
I20260925 15:07:50.245990 13018 tablet_bootstrap.cc:654] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.247150 13020 tablet_bootstrap.cc:654] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:50.254761 13042 ts_tablet_manager.cc:1918] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:50.254974 13042 log.cc:1196] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/91b2beceae764a3390c91669580ad643
I20260925 15:07:50.255470 13042 ts_tablet_manager.cc:1939] T 91b2beceae764a3390c91669580ad643 P ad0319b5273145b8ba8cf56f74086d1c: Deleting consensus metadata
I20260925 15:07:50.254761 13020 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: No bootstrap required, opened a new log
I20260925 15:07:50.255810 13020 ts_tablet_manager.cc:1397] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 15:07:50.256598 13020 raft_consensus.cc:348] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.256958 13020 raft_consensus.cc:374] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.257074 13020 raft_consensus.cc:729] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad0319b5273145b8ba8cf56f74086d1c, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.257382 13020 consensus_queue.cc:260] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.258888 13020 ts_tablet_manager.cc:1428] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 15:07:50.260334 12494 catalog_manager.cc:4841] TS ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873): tablet 91b2beceae764a3390c91669580ad643 (table test-table [id=0334fb751d434a23b1f265df1035f03b]) successfully deleted
I20260925 15:07:50.261081 13041 ts_tablet_manager.cc:1905] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:50.263037 13018 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: No bootstrap required, opened a new log
I20260925 15:07:50.263196 13018 ts_tablet_manager.cc:1397] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.008s
I20260925 15:07:50.263972 13018 raft_consensus.cc:348] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.264225 13018 raft_consensus.cc:374] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.264303 13018 raft_consensus.cc:729] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.264542 13018 consensus_queue.cc:260] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.265044 13018 ts_tablet_manager.cc:1428] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 15:07:50.270671 13019 tablet_bootstrap.cc:492] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: No bootstrap required, opened a new log
I20260925 15:07:50.270843 13019 ts_tablet_manager.cc:1397] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.006s
I20260925 15:07:50.271754 13019 raft_consensus.cc:348] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.271998 13019 raft_consensus.cc:374] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:50.272076 13019 raft_consensus.cc:729] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87541fdc92a478fb1e45b35f01baf8d, State: Initialized, Role: FOLLOWER
I20260925 15:07:50.272397 13019 consensus_queue.cc:260] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.272907 13019 ts_tablet_manager.cc:1428] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 15:07:50.273381 13041 ts_tablet_manager.cc:1918] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:07:50.273561 13041 log.cc:1196] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/91b2beceae764a3390c91669580ad643
I20260925 15:07:50.274096 13041 ts_tablet_manager.cc:1939] T 91b2beceae764a3390c91669580ad643 P b87541fdc92a478fb1e45b35f01baf8d: Deleting consensus metadata
I20260925 15:07:50.276122 12494 catalog_manager.cc:4841] TS b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947): tablet 91b2beceae764a3390c91669580ad643 (table test-table [id=0334fb751d434a23b1f265df1035f03b]) successfully deleted
W20260925 15:07:50.276511 12494 catalog_manager.cc:4808] TS b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947): delete failed for tablet 91b2beceae764a3390c91669580ad643 because the tablet was not found. No further retry: Not found: Tablet not found: 91b2beceae764a3390c91669580ad643
I20260925 15:07:50.566610 13026 raft_consensus.cc:482] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:07:50.566807 13026 raft_consensus.cc:504] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.567483 13026 leader_election.cc:288] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:50.567929 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5f368f2aa440618513ec384eaba9a1" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c" is_pre_election: true
I20260925 15:07:50.568022 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5f368f2aa440618513ec384eaba9a1" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d" is_pre_election: true
I20260925 15:07:50.568176 12804 raft_consensus.cc:2454] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:50.568222 12948 raft_consensus.cc:2454] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 0.
I20260925 15:07:50.568624 12597 leader_election.cc:302] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad0319b5273145b8ba8cf56f74086d1c, ae7b58600bbe4cf38c12cd53518ba26d; no voters:
I20260925 15:07:50.568905 13026 raft_consensus.cc:2781] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:07:50.569046 13026 raft_consensus.cc:482] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:07:50.569136 13026 raft_consensus.cc:3037] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.571072 13026 raft_consensus.cc:504] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.571565 13026 leader_election.cc:288] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Requested vote from peers ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873), b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947)
I20260925 15:07:50.572000 12804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5f368f2aa440618513ec384eaba9a1" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad0319b5273145b8ba8cf56f74086d1c"
I20260925 15:07:50.572086 12948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f5f368f2aa440618513ec384eaba9a1" candidate_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b87541fdc92a478fb1e45b35f01baf8d"
I20260925 15:07:50.572223 12804 raft_consensus.cc:3037] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.572258 12948 raft_consensus.cc:3037] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:50.575054 12804 raft_consensus.cc:2454] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:50.575069 12948 raft_consensus.cc:2454] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae7b58600bbe4cf38c12cd53518ba26d in term 1.
I20260925 15:07:50.575480 12597 leader_election.cc:302] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ae7b58600bbe4cf38c12cd53518ba26d, b87541fdc92a478fb1e45b35f01baf8d; no voters:
I20260925 15:07:50.575783 13026 raft_consensus.cc:2781] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:50.575942 13026 raft_consensus.cc:686] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Becoming Leader. State: Replica: ae7b58600bbe4cf38c12cd53518ba26d, State: Running, Role: LEADER
I20260925 15:07:50.576205 13026 consensus_queue.cc:237] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } }
I20260925 15:07:50.578676 12507 catalog_manager.cc:5495] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d reported cstate change: term changed from 0 to 1, leader changed from <none> to ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193). New cstate: current_term: 1 leader_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ae7b58600bbe4cf38c12cd53518ba26d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38829 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 } health_report { overall_health: UNKNOWN } } }
I20260925 15:07:50.628625 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.624168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50763 (local address 127.4.205.193:38829)
0925 15:07:50.624354 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.624358 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.624596 (+ 238us) server_negotiation.cc:407] Connection header received
0925 15:07:50.624714 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.624717 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.624771 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.624844 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:07:50.625207 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.626019 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.627289 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.627694 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.627777 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.628091 (+ 314us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:07:50.628171 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:07:50.628229 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.628367 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.628419 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:07:50.628791 13103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.623992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38829 (local address 127.4.205.193:50763)
0925 15:07:50.624504 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:50.624519 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:50.624615 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:50.624863 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:50.624866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:50.624874 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 15:07:50.625059 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.625066 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.626126 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.626130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.627178 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:50.627188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.627841 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.627845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:50.627947 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.627977 (+ 30us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 15:07:50.628262 (+ 285us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 15:07:50.628271 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:07:50.628337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 15:07:50.628434 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":2,"thread_start_us":266,"threads_started":1}
I20260925 15:07:50.636152 12804 raft_consensus.cc:1261] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:50.636186 12948 raft_consensus.cc:1261] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Refusing update from remote peer ae7b58600bbe4cf38c12cd53518ba26d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:07:50.636631 13067 consensus_queue.cc:1035] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b87541fdc92a478fb1e45b35f01baf8d" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:50.636904 13029 consensus_queue.cc:1035] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad0319b5273145b8ba8cf56f74086d1c" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:07:50.674871 12507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:50.675084 12507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43708:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:07:50.677630 12507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 54741c469c854cc5bc017bfb6375394a: Sending DeleteTablet for 3 replicas of tablet 9f5f368f2aa440618513ec384eaba9a1
I20260925 15:07:50.678284 12645 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60758
I20260925 15:07:50.678408 12784 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49920
I20260925 15:07:50.678469 12922 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f5f368f2aa440618513ec384eaba9a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:07:50 UTC) from {username='oryx', principal='oryx/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:59934
I20260925 15:07:50.678557 13043 tablet_replica.cc:330] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: stopping tablet replica
I20260925 15:07:50.678707 13041 tablet_replica.cc:330] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: stopping tablet replica
I20260925 15:07:50.678731 13043 raft_consensus.cc:2229] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:07:50.678834 13042 tablet_replica.cc:330] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: stopping tablet replica
I20260925 15:07:50.678887 13041 raft_consensus.cc:2229] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:50.679035 13043 raft_consensus.cc:2258] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.679049 13042 raft_consensus.cc:2229] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:07:50.679066 13041 raft_consensus.cc:2258] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.679205 13042 raft_consensus.cc:2258] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:07:50.679257 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12579
I20260925 15:07:50.680192 13043 ts_tablet_manager.cc:1905] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:50.680655 13041 ts_tablet_manager.cc:1905] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:50.681919 13042 ts_tablet_manager.cc:1905] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:07:50.686638 13043 ts_tablet_manager.cc:1918] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:50.686769 13043 log.cc:1196] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/9f5f368f2aa440618513ec384eaba9a1
I20260925 15:07:50.687237 13043 ts_tablet_manager.cc:1939] T 9f5f368f2aa440618513ec384eaba9a1 P ae7b58600bbe4cf38c12cd53518ba26d: Deleting consensus metadata
I20260925 15:07:50.688299 13042 ts_tablet_manager.cc:1918] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:50.688453 13042 log.cc:1196] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/9f5f368f2aa440618513ec384eaba9a1
I20260925 15:07:50.688833 13042 ts_tablet_manager.cc:1939] T 9f5f368f2aa440618513ec384eaba9a1 P ad0319b5273145b8ba8cf56f74086d1c: Deleting consensus metadata
I20260925 15:07:50.689327 13041 ts_tablet_manager.cc:1918] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 15:07:50.689503 13041 log.cc:1196] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/9f5f368f2aa440618513ec384eaba9a1
I20260925 15:07:50.689846 12494 catalog_manager.cc:4841] TS ae7b58600bbe4cf38c12cd53518ba26d (127.4.205.193:38829): tablet 9f5f368f2aa440618513ec384eaba9a1 (table test-table [id=0fd4af541b9942caa720b8386ca4d29c]) successfully deleted
I20260925 15:07:50.690080 12494 catalog_manager.cc:4841] TS ad0319b5273145b8ba8cf56f74086d1c (127.4.205.194:37873): tablet 9f5f368f2aa440618513ec384eaba9a1 (table test-table [id=0fd4af541b9942caa720b8386ca4d29c]) successfully deleted
I20260925 15:07:50.690375 13041 ts_tablet_manager.cc:1939] T 9f5f368f2aa440618513ec384eaba9a1 P b87541fdc92a478fb1e45b35f01baf8d: Deleting consensus metadata
I20260925 15:07:50.691259 12494 catalog_manager.cc:4841] TS b87541fdc92a478fb1e45b35f01baf8d (127.4.205.195:44947): tablet 9f5f368f2aa440618513ec384eaba9a1 (table test-table [id=0fd4af541b9942caa720b8386ca4d29c]) successfully deleted
Sep 25 15:07:50 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.193@KRBTEST.COM
I20260925 15:07:50.702826 13013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.680414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60780 (local address 127.4.205.193:38829)
0925 15:07:50.680591 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.680597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.690104 (+ 9507us) server_negotiation.cc:407] Connection header received
0925 15:07:50.690284 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.690290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.690382 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.690491 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.691765 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.692469 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.693602 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.693947 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.694036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.697893 (+ 3857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.697906 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.697921 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.697955 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.700579 (+ 2624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.701110 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.701117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.701119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.701235 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.701546 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.701549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.701551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.701929 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.702151 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.702347 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.702506 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:07:50.703461 12710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:50.844307 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12579
I20260925 15:07:50.868992 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12719
Sep 25 15:07:50 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.194@KRBTEST.COM
I20260925 15:07:50.883416 13109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:50.869797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49944 (local address 127.4.205.194:37873)
0925 15:07:50.870210 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:50.870215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:50.870231 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:50.870283 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:50.870286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:50.870341 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:50.870432 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:50.871706 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.872372 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.873660 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:50.874024 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:50.874112 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:50.877983 (+ 3871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:50.877995 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:50.878009 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:50.878041 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:50.880556 (+ 2515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.881156 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.881163 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.881165 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.881315 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:50.881819 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:50.881827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:50.881831 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:50.882173 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:50.882484 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:50.882802 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:07:50.883066 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":211,"threads_started":1}
I20260925 15:07:50.884088 12849 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.003970 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12719
I20260925 15:07:51.025049 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12858
Sep 25 15:07:51 dist-test-slave-64fx krb5kdc[12463](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.4.205.195@KRBTEST.COM
I20260925 15:07:51.039459 13111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:51.025831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.4.205.195:44947)
0925 15:07:51.026331 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:51.026335 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:51.026349 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:07:51.026407 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:51.026410 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:51.026465 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:51.026590 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:51.027821 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.028478 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.029848 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:51.030273 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:51.030412 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:51.034502 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:51.034515 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:51.034531 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:51.034562 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:51.037143 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.037759 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.037766 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.037769 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.037879 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:51.038199 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:51.038202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:51.038204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:51.038527 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:51.038727 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:51.038964 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 15:07:51.039197 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":249,"threads_started":1}
I20260925 15:07:51.040165 12988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.134161 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12858
I20260925 15:07:51.150853 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 12476
I20260925 15:07:51.151975 12548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:51.262153 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 12476
2026-09-25T15:07:51Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6456 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:51 dist-test-slave-64fx krb5kdc[13116](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:51 dist-test-slave-64fx krb5kdc[13116](info): set up 2 sockets
Sep 25 15:07:51 dist-test-slave-64fx krb5kdc[13116](info): commencing operation
krb5kdc: starting...
W20260925 15:07:52.318337 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:52 dist-test-slave-64fx krb5kdc[13116](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:52.508423 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39219
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:52.802218 13129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:52.802457 13129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:52.802544 13129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:52.808676 13129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:52.808753 13129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:52.808795 13129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:52.808838 13129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:52.808873 13129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:52.817011 13129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:52.817941 13129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:52.818966 13129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:52.829698 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:52.829756 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:52.829722 13137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:52.830367 13129 server_base.cc:1034] running on GCE node
I20260925 15:07:52.830958 13129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:52.832108 13129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:52.833326 13129 hybrid_clock.cc:648] HybridClock initialized: now 1790348872833275 us; error 48 us; skew 500 ppm
I20260925 15:07:52.833679 13129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T15:07:52Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1637 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:07:52 dist-test-slave-64fx krb5kdc[13143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:07:52 dist-test-slave-64fx krb5kdc[13143](info): set up 2 sockets
Sep 25 15:07:52 dist-test-slave-64fx krb5kdc[13143](info): commencing operation
krb5kdc: starting...
W20260925 15:07:53.954481 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.005s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:07:54 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:07:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:07:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:07:54.153260 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42671
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:54.460189 13156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:54.460525 13156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:54.460678 13156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:54.467671 13156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:54.467803 13156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:54.467907 13156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:54.468006 13156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:54.468088 13156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:54.477375 13156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:42671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:54.478591 13156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:54.479851 13156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:54.490911 13164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:54.491003 13161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:54.490979 13162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:54.491748 13156 server_base.cc:1034] running on GCE node
I20260925 15:07:54.492411 13156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:54.493700 13156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:54.495092 13156 hybrid_clock.cc:648] HybridClock initialized: now 1790348874495043 us; error 36 us; skew 500 ppm
I20260925 15:07:54.495464 13156 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:54 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:54.498407 13156 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:54.499745 13156 webserver.cc:460] Webserver started at http://127.4.205.254:38281/ using document root <none> and password file <none>
I20260925 15:07:54.500351 13156 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:54.500470 13156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:54.500761 13156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:54.503489 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "f0f91934a39e41c0bc791d733537da5a"
format_stamp: "Formatted at 2026-09-25 15:07:54 on dist-test-slave-64fx"
server_key: "0eaf13ed4f877f26a9f98a51ed4157a5"
server_key_iv: "c15d9ec39e11c9f03cd43002be02fbf3"
server_key_version: "encryptionkey@0"
I20260925 15:07:54.504194 13156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "f0f91934a39e41c0bc791d733537da5a"
format_stamp: "Formatted at 2026-09-25 15:07:54 on dist-test-slave-64fx"
server_key: "0eaf13ed4f877f26a9f98a51ed4157a5"
server_key_iv: "c15d9ec39e11c9f03cd43002be02fbf3"
server_key_version: "encryptionkey@0"
I20260925 15:07:54.508668 13156 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260925 15:07:54.511721 13171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:54.512893 13156 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:07:54.513069 13156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "f0f91934a39e41c0bc791d733537da5a"
format_stamp: "Formatted at 2026-09-25 15:07:54 on dist-test-slave-64fx"
server_key: "0eaf13ed4f877f26a9f98a51ed4157a5"
server_key_iv: "c15d9ec39e11c9f03cd43002be02fbf3"
server_key_version: "encryptionkey@0"
I20260925 15:07:54.513260 13156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:54.528312 13156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:54.531787 13156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:54.532042 13156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:54.548939 13156 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:42671
I20260925 15:07:54.548960 13222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:42671 every 8 connection(s)
I20260925 15:07:54.550182 13156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:54.550544 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13156
I20260925 15:07:54.550734 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:07:54.551565 4919 external_mini_cluster.cc:1428] Setting key 248539c765ad550c83d3a07bc76b7d8f
I20260925 15:07:54.554268 13223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:07:54 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:54.565955 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Bootstrap starting.
I20260925 15:07:54.569675 13223 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Neither blocks nor log segments found. Creating new log.
I20260925 15:07:54.570906 13223 log.cc:824] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:54.574683 13223 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: No bootstrap required, opened a new log
I20260925 15:07:54.575191 13226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:54.553430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.4.205.254:42671)
0925 15:07:54.554244 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:54.554263 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:54.554312 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:07:54.556505 (+ 2193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:54.556528 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:54.556973 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:54.557383 (+ 410us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:54.558645 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:54.560603 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:54.561838 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:54.562816 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:54.563005 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:54.566615 (+ 3610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:54.566625 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:54.566640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:54.566677 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:54.570793 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:54.571696 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:54.571708 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:54.571737 (+ 29us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:54.571973 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:54.572422 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:54.572428 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:54.572434 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:54.573016 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:54.573297 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:54.573829 (+ 532us) server_negotiation.cc:299] Negotiation successful
0925 15:07:54.574240 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":319,"threads_started":1}
I20260925 15:07:54.582895 13223 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:54.583231 13223 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:07:54.583351 13223 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f91934a39e41c0bc791d733537da5a, State: Initialized, Role: FOLLOWER
I20260925 15:07:54.583920 13223 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:54.584144 13223 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:54.584285 13223 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:54.584472 13223 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:07:54.586499 13223 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:54.587025 13223 leader_election.cc:302] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0f91934a39e41c0bc791d733537da5a; no voters:
I20260925 15:07:54.587567 13223 leader_election.cc:288] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:07:54.587652 13228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:07:54.588328 13228 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 LEADER]: Becoming Leader. State: Replica: f0f91934a39e41c0bc791d733537da5a, State: Running, Role: LEADER
I20260925 15:07:54.588836 13228 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:54.589479 13223 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:54.591372 13230 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f91934a39e41c0bc791d733537da5a. Latest consensus state: current_term: 1 leader_uuid: "f0f91934a39e41c0bc791d733537da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } } }
I20260925 15:07:54.591796 13230 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:54.592265 13229 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0f91934a39e41c0bc791d733537da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } } }
I20260925 15:07:54.592518 13229 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:54.594424 13237 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:54.598842 13237 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:54.606056 13237 catalog_manager.cc:1347] Generated new cluster ID: 3518f60e9eae41a19da488388cbd58b9
I20260925 15:07:54.606149 13237 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:54.625551 13237 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:07:54.626677 13237 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:54.633605 13237 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Generated new TSK 0
I20260925 15:07:54.634447 13237 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:07:54.693372 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:54.996300 13249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:54.996548 13249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:54.996695 13249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:55.003396 13249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:55.003482 13249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:55.003695 13249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:55.012652 13249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:55.013863 13249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:55.015086 13249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:55.027397 13255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:55.027626 13254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:55.027593 13257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:55.028544 13249 server_base.cc:1034] running on GCE node
I20260925 15:07:55.029054 13249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:55.029884 13249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:55.031095 13249 hybrid_clock.cc:648] HybridClock initialized: now 1790348875031055 us; error 52 us; skew 500 ppm
I20260925 15:07:55.031376 13249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:55 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:55.042086 13249 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:07:55.043500 13249 webserver.cc:460] Webserver started at http://127.4.205.193:32909/ using document root <none> and password file <none>
I20260925 15:07:55.044124 13249 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:55.044211 13249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:55.044430 13249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:55.047103 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "79d1570fe27d4496b23ba6dace55854a"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "c91a7837d8a503471366f931de1bd9f6"
server_key_iv: "bcf9a7a3af3f1a2e43853f5b4b768936"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.047770 13249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "79d1570fe27d4496b23ba6dace55854a"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "c91a7837d8a503471366f931de1bd9f6"
server_key_iv: "bcf9a7a3af3f1a2e43853f5b4b768936"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.052310 13249 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:55.055318 13264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.056492 13249 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:07:55.056619 13249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "79d1570fe27d4496b23ba6dace55854a"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "c91a7837d8a503471366f931de1bd9f6"
server_key_iv: "bcf9a7a3af3f1a2e43853f5b4b768936"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.056864 13249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:55.098070 13249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:55.101807 13249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:55.102035 13249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:55.102892 13249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:55.104076 13249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:55.104143 13249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.104295 13249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:55.104351 13249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.135360 13249 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:39613
I20260925 15:07:55.135434 13376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:39613 every 8 connection(s)
I20260925 15:07:55.136533 13249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:55.143049 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13249
I20260925 15:07:55.143237 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:07:55.144017 4919 external_mini_cluster.cc:1428] Setting key e330521df28f296d394cd31bf431f3dc
Sep 25 15:07:55 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:55.156503 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:55.138831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57087 (local address 127.4.205.254:42671)
0925 15:07:55.139252 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:55.139258 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:55.140175 (+ 917us) server_negotiation.cc:407] Connection header received
0925 15:07:55.141910 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:55.141914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:55.141973 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:55.142064 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:55.143868 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.144672 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.146697 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.147057 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.147146 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:55.151041 (+ 3895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:55.151052 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:55.151055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:55.151084 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:55.153625 (+ 2541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:55.154335 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:55.154341 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:55.154347 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:55.154462 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:55.154874 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:55.154880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:55.154884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:55.155133 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:55.155287 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:55.155939 (+ 652us) server_negotiation.cc:299] Negotiation successful
0925 15:07:55.156149 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":195,"threads_started":1}
I20260925 15:07:55.157606 13380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:55.139403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:57087)
0925 15:07:55.140026 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:55.140062 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:55.141683 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:55.142247 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:55.142253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:55.142889 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:55.143696 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:55.143723 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.144852 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.144857 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:55.146546 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:55.146564 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.147263 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.147272 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:55.147591 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:55.148268 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:55.148298 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:55.150833 (+ 2535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:55.153755 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:55.153765 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:55.153780 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:55.154190 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:55.154570 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:55.154576 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:55.154581 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:55.154759 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:55.155272 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:55.155280 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:55.155579 (+ 299us) client_negotiation.cc:769] Sending connection context
0925 15:07:55.155834 (+ 255us) client_negotiation.cc:240] Negotiation successful
0925 15:07:55.156318 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":251,"threads_started":1}
I20260925 15:07:55.159147 13377 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42671
I20260925 15:07:55.159487 13377 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:55.160342 13377 heartbeater.cc:511] Master 127.4.205.254:42671 requested a full tablet report, sending...
I20260925 15:07:55.162799 13188 ts_manager.cc:194] Registered new tserver with Master: 79d1570fe27d4496b23ba6dace55854a (127.4.205.193:39613)
I20260925 15:07:55.164929 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:57087
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:07:55.192444 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:55.501785 13383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:55.502041 13383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:55.502132 13383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:55.509054 13383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:55.509146 13383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:55.509256 13383 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:55.518605 13383 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:55.519706 13383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:55.520891 13383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:55.533211 13391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:55.533255 13389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:55.533293 13388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:55.534183 13383 server_base.cc:1034] running on GCE node
I20260925 15:07:55.534754 13383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:55.535655 13383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:55.536865 13383 hybrid_clock.cc:648] HybridClock initialized: now 1790348875536810 us; error 47 us; skew 500 ppm
I20260925 15:07:55.537151 13383 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:55 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:55.540163 13383 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:07:55.541502 13383 webserver.cc:460] Webserver started at http://127.4.205.194:37367/ using document root <none> and password file <none>
I20260925 15:07:55.542193 13383 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:55.542351 13383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:55.542646 13383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:55.545351 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "9dbfae420a1744d3b02e4c135f4bb171"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "93edb4ef7cae4f36807da47f6a66ee44"
server_key_iv: "5f020d4ef5eaf3c020773b202c4dc421"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.546135 13383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "9dbfae420a1744d3b02e4c135f4bb171"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "93edb4ef7cae4f36807da47f6a66ee44"
server_key_iv: "5f020d4ef5eaf3c020773b202c4dc421"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.551041 13383 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:07:55.554286 13398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.555600 13383 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:55.555760 13383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "9dbfae420a1744d3b02e4c135f4bb171"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "93edb4ef7cae4f36807da47f6a66ee44"
server_key_iv: "5f020d4ef5eaf3c020773b202c4dc421"
server_key_version: "encryptionkey@0"
I20260925 15:07:55.555955 13383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:55.568632 13383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:55.572475 13383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:55.572800 13383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:55.573796 13383 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:55.575028 13383 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:55.575130 13383 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.575276 13383 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:55.575358 13383 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:55.606586 13383 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:36953
I20260925 15:07:55.606629 13510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:36953 every 8 connection(s)
I20260925 15:07:55.607867 13383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:55.609485 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13383
I20260925 15:07:55.609694 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:07:55.610281 4919 external_mini_cluster.cc:1428] Setting key b9c79ec55684651caa578e55404cc46e
Sep 25 15:07:55 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348875, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:55.628154 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:55.610649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53007 (local address 127.4.205.254:42671)
0925 15:07:55.610813 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:55.610837 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:55.612028 (+ 1191us) server_negotiation.cc:407] Connection header received
0925 15:07:55.613868 (+ 1840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:55.613874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:55.613964 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:55.614068 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:55.615442 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.616204 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.617861 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.618285 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.618417 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:55.621691 (+ 3274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:55.621699 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:55.621707 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:55.621750 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:55.625208 (+ 3458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:55.625929 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:55.625937 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:55.625943 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:55.626097 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:55.626483 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:55.626491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:55.626497 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:55.626866 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:55.627014 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:55.627615 (+ 601us) server_negotiation.cc:299] Negotiation successful
0925 15:07:55.627866 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:07:55.629261 13513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:55.611155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:53007)
0925 15:07:55.611884 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:55.611925 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:55.613627 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:55.614282 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:55.614290 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:55.614762 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:55.615285 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:55.615300 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.616368 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.616372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:55.617728 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:55.617739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:55.618503 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:55.618510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:55.618719 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:55.619362 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:55.619384 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:55.621441 (+ 2057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:55.625343 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:55.625349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:55.625360 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:55.625793 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:55.626204 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:55.626207 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:55.626210 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:55.626363 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:55.626987 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:55.626992 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:55.627269 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 15:07:55.627466 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 15:07:55.627796 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":315,"threads_started":1}
I20260925 15:07:55.630736 13511 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42671
I20260925 15:07:55.631063 13511 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:07:55.631918 13511 heartbeater.cc:511] Master 127.4.205.254:42671 requested a full tablet report, sending...
I20260925 15:07:55.633385 13188 ts_manager.cc:194] Registered new tserver with Master: 9dbfae420a1744d3b02e4c135f4bb171 (127.4.205.194:36953)
I20260925 15:07:55.634300 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:53007
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:07:55.660894 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:55.973610 13516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:55.973927 13516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:55.974095 13516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:55.981119 13516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:55.981277 13516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:55.981446 13516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:55.990842 13516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:55.992067 13516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:55.993328 13516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:56.005635 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:56.005672 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:56.005712 13524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:56.006534 13516 server_base.cc:1034] running on GCE node
I20260925 15:07:56.007068 13516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:56.007936 13516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:56.009160 13516 hybrid_clock.cc:648] HybridClock initialized: now 1790348876009126 us; error 46 us; skew 500 ppm
I20260925 15:07:56.009469 13516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:56 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348876, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:56.012521 13516 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:07:56.013857 13516 webserver.cc:460] Webserver started at http://127.4.205.195:38575/ using document root <none> and password file <none>
I20260925 15:07:56.014460 13516 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:56.014554 13516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:56.014786 13516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:07:56.017493 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "638a244ed56f4c3a908c985b1a48cdaf"
format_stamp: "Formatted at 2026-09-25 15:07:56 on dist-test-slave-64fx"
server_key: "421851a44a1ef3edc881eac99622a867"
server_key_iv: "6050018625a6c46f95938f31e0eff142"
server_key_version: "encryptionkey@0"
I20260925 15:07:56.018270 13516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "638a244ed56f4c3a908c985b1a48cdaf"
format_stamp: "Formatted at 2026-09-25 15:07:56 on dist-test-slave-64fx"
server_key: "421851a44a1ef3edc881eac99622a867"
server_key_iv: "6050018625a6c46f95938f31e0eff142"
server_key_version: "encryptionkey@0"
I20260925 15:07:56.022900 13516 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:07:56.025998 13531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:56.027184 13516 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 15:07:56.027312 13516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "638a244ed56f4c3a908c985b1a48cdaf"
format_stamp: "Formatted at 2026-09-25 15:07:56 on dist-test-slave-64fx"
server_key: "421851a44a1ef3edc881eac99622a867"
server_key_iv: "6050018625a6c46f95938f31e0eff142"
server_key_version: "encryptionkey@0"
I20260925 15:07:56.027462 13516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:56.039842 13516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:56.043715 13516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:56.043923 13516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:56.044705 13516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:56.046022 13516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:56.046092 13516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:56.046252 13516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:56.046316 13516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:56.077975 13516 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:35885
I20260925 15:07:56.078020 13643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:35885 every 8 connection(s)
I20260925 15:07:56.079161 13516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:56.080233 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13516
I20260925 15:07:56.080408 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:07:56.080922 4919 external_mini_cluster.cc:1428] Setting key 68327b8e6034d9c7e2abc0e3bc08824d
Sep 25 15:07:56 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348876, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:07:56.097898 13379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:56.082912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:49667 (local address 127.4.205.254:42671)
0925 15:07:56.083121 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:56.083127 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:56.083362 (+ 235us) server_negotiation.cc:407] Connection header received
0925 15:07:56.085325 (+ 1963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:56.085330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:56.085448 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:56.085548 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:56.086931 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.087652 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.089098 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.089522 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.089697 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:56.092739 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:56.092743 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:56.092746 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:56.092772 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:56.095227 (+ 2455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.095981 (+ 754us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.095985 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.095987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.096087 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.096572 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.096575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.096577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.096807 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:56.096913 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:56.097468 (+ 555us) server_negotiation.cc:299] Negotiation successful
0925 15:07:56.097665 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:07:56.098719 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:56.082476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:49667)
0925 15:07:56.083229 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:56.083275 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:56.085082 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:56.085778 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:56.085785 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:56.086263 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:56.086774 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:56.086790 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.087786 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.087792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:56.088962 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:56.088972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.089769 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.089778 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:56.090010 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:56.090537 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:56.090557 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:56.092577 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:07:56.095392 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:56.095402 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:56.095415 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:56.095878 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:56.096233 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:07:56.096238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:07:56.096241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:07:56.096478 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:07:56.096908 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:07:56.096913 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:07:56.097188 (+ 275us) client_negotiation.cc:769] Sending connection context
0925 15:07:56.097390 (+ 202us) client_negotiation.cc:240] Negotiation successful
0925 15:07:56.097791 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":289,"threads_started":1}
I20260925 15:07:56.099956 13644 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42671
I20260925 15:07:56.100275 13644 heartbeater.cc:464] Registering TS with master...
I20260925 15:07:56.101044 13644 heartbeater.cc:511] Master 127.4.205.254:42671 requested a full tablet report, sending...
I20260925 15:07:56.102334 13188 ts_manager.cc:194] Registered new tserver with Master: 638a244ed56f4c3a908c985b1a48cdaf (127.4.205.195:35885)
I20260925 15:07:56.103075 13188 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:49667
I20260925 15:07:56.108172 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
Entry for principal oryx/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab.
WARNING: no policy specified for oryx/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:07:56.168045 13377 heartbeater.cc:503] Master 127.4.205.254:42671 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
Entry for principal oryx/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab.
WARNING: no policy specified for oryx/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
Entry for principal oryx/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab.
I20260925 15:07:56.230481 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13249
Sep 25 15:07:56 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:07:56.245445 13654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:56.231625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46980 (local address 127.4.205.193:39613)
0925 15:07:56.232247 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:56.232255 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:56.232282 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:07:56.232352 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:56.232359 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:56.232560 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:56.232698 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:56.234043 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.234997 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.236174 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.236714 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.236801 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:56.240610 (+ 3809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:56.240621 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:56.240634 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:56.240665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:56.243291 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.243853 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.243859 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.243861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.243984 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.244264 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.244267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.244269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.244550 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:56.244753 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:56.244973 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:07:56.245186 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":341,"threads_started":1}
I20260925 15:07:56.246644 13372 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:56.330142 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13249
I20260925 15:07:56.346136 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13383
Sep 25 15:07:56 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:07:56.360880 13656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:56.346946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36494 (local address 127.4.205.194:36953)
0925 15:07:56.347492 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:56.347499 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:56.347522 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:07:56.347581 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:56.347587 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:56.347794 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:56.347926 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:56.349026 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.350101 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.351256 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.351756 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.351854 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:56.355671 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:56.355680 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:56.355692 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:56.355716 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:56.358278 (+ 2562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.358822 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.358829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.358831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.359006 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.359459 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.359466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.359472 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.359826 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:56.360117 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:56.360331 (+ 214us) server_negotiation.cc:299] Negotiation successful
0925 15:07:56.360508 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":303,"threads_started":1}
I20260925 15:07:56.362111 13506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:56.463549 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13383
I20260925 15:07:56.479334 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13516
Sep 25 15:07:56 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:07:56.494678 13658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:56.480398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.4.205.195:35885)
0925 15:07:56.481086 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:56.481094 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:56.481119 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:07:56.481173 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:56.481180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:56.481418 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:56.481540 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:56.482793 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.483702 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.484830 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:56.485351 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:56.485439 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:56.489367 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:56.489378 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:56.489392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:56.489420 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:56.492208 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.492852 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.492859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.492861 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.492979 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:56.493318 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:56.493321 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:56.493323 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:56.493734 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:56.493963 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:56.494134 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:07:56.494305 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":395,"threads_started":1}
I20260925 15:07:56.495872 13639 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:56.573602 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13516
I20260925 15:07:56.588663 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13156
I20260925 15:07:56.589828 13218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:07:56.686944 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13156
I20260925 15:07:56.706390 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42671
--webserver_interface=127.4.205.254
--webserver_port=38281
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:42671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:07:57.014246 13661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:57.014495 13661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:57.014664 13661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:57.021467 13661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:07:57.021605 13661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:57.021713 13661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:57.021806 13661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:07:57.021890 13661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:07:57.031190 13661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:42671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=38281
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:57.032402 13661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:57.033735 13661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:57.044999 13669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:57.045032 13667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:57.045091 13666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:57.045790 13661 server_base.cc:1034] running on GCE node
I20260925 15:07:57.046526 13661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:57.047791 13661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:57.049011 13661 hybrid_clock.cc:648] HybridClock initialized: now 1790348877048970 us; error 55 us; skew 500 ppm
I20260925 15:07:57.049348 13661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:57 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348877, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:57.052336 13661 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:07:57.053746 13661 webserver.cc:460] Webserver started at http://127.4.205.254:38281/ using document root <none> and password file <none>
I20260925 15:07:57.054348 13661 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:57.054450 13661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:57.059204 13661 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 15:07:57.061813 13676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.062978 13661 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:07:57.063097 13661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "f0f91934a39e41c0bc791d733537da5a"
format_stamp: "Formatted at 2026-09-25 15:07:54 on dist-test-slave-64fx"
server_key: "0eaf13ed4f877f26a9f98a51ed4157a5"
server_key_iv: "c15d9ec39e11c9f03cd43002be02fbf3"
server_key_version: "encryptionkey@0"
I20260925 15:07:57.063995 13661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:57.084661 13661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:57.088059 13661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:57.088258 13661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:57.105826 13661 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:42671
I20260925 15:07:57.105845 13727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:42671 every 8 connection(s)
I20260925 15:07:57.107095 13661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:07:57.113129 13728 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:07:57.114903 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13661
I20260925 15:07:57.115629 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:39613
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:42671
--webserver_port=32909
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:07:57.116096 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Bootstrap starting.
I20260925 15:07:57.133292 13728 log.cc:824] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:07:57.140776 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:07:57.141224 13728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Bootstrap complete.
I20260925 15:07:57.149710 13728 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:57.150491 13728 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0f91934a39e41c0bc791d733537da5a, State: Initialized, Role: FOLLOWER
I20260925 15:07:57.151032 13728 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:57.151244 13728 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:07:57.151374 13728 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:07:57.151541 13728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:07:57.154978 13728 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:57.155408 13728 leader_election.cc:302] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0f91934a39e41c0bc791d733537da5a; no voters:
I20260925 15:07:57.156004 13728 leader_election.cc:288] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:07:57.156126 13732 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:07:57.157133 13732 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [term 2 LEADER]: Becoming Leader. State: Replica: f0f91934a39e41c0bc791d733537da5a, State: Running, Role: LEADER
I20260925 15:07:57.157825 13732 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } }
I20260925 15:07:57.158013 13728 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:07:57.160231 13734 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0f91934a39e41c0bc791d733537da5a. Latest consensus state: current_term: 2 leader_uuid: "f0f91934a39e41c0bc791d733537da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } } }
I20260925 15:07:57.160650 13734 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:57.161510 13733 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f0f91934a39e41c0bc791d733537da5a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0f91934a39e41c0bc791d733537da5a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42671 } } }
I20260925 15:07:57.161883 13733 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a [sys.catalog]: This master's current role is: LEADER
I20260925 15:07:57.170795 13739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:07:57.176982 13739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:07:57.178640 13739 catalog_manager.cc:1259] Loaded cluster ID: 3518f60e9eae41a19da488388cbd58b9
I20260925 15:07:57.178789 13739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:07:57.183380 13739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:07:57.185153 13739 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f0f91934a39e41c0bc791d733537da5a: Loaded TSK: 0
I20260925 15:07:57.186903 13739 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:07:57.453488 13730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:57.453837 13730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:57.453995 13730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:57.460975 13730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:57.461114 13730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:57.461292 13730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:07:57.470870 13730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.193.keytab
--principal=oryx/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:39613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=32909
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:57.472092 13730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:57.473332 13730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:57.487411 13754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:57.487411 13755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:57.488277 13730 server_base.cc:1034] running on GCE node
W20260925 15:07:57.488552 13757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:57.489197 13730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:57.490195 13730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:57.491405 13730 hybrid_clock.cc:648] HybridClock initialized: now 1790348877491360 us; error 39 us; skew 500 ppm
I20260925 15:07:57.491700 13730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:57 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348877, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:57.494694 13730 init.cc:377] Logged in from keytab as oryx/127.4.205.193@KRBTEST.COM (short username oryx)
I20260925 15:07:57.496119 13730 webserver.cc:460] Webserver started at http://127.4.205.193:32909/ using document root <none> and password file <none>
I20260925 15:07:57.496735 13730 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:57.496865 13730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:57.501927 13730 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:07:57.504602 13764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.506035 13730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:57.506181 13730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "79d1570fe27d4496b23ba6dace55854a"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "c91a7837d8a503471366f931de1bd9f6"
server_key_iv: "bcf9a7a3af3f1a2e43853f5b4b768936"
server_key_version: "encryptionkey@0"
I20260925 15:07:57.507072 13730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:57.518584 13730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:57.522528 13730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:57.522796 13730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:57.523754 13730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:57.525147 13730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:57.525238 13730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.525377 13730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:57.525452 13730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.555923 13730 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:39613
I20260925 15:07:57.556000 13876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:39613 every 8 connection(s)
I20260925 15:07:57.557147 13730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:07:57.565049 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13730
I20260925 15:07:57.565659 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:36953
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:42671
--webserver_port=37367
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:57 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:57 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:57.899003 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:57.899281 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:57.899441 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:57.906436 13881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:57.906584 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:57.906749 13881 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:07:57.916057 13881 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.194.keytab
--principal=oryx/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:36953
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=37367
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:57.917291 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:57.918602 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:57.935438 13888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:57.935462 13887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:57.936340 13890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:57.936806 13881 server_base.cc:1034] running on GCE node
I20260925 15:07:57.937778 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:57.939276 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:57.940564 13881 hybrid_clock.cc:648] HybridClock initialized: now 1790348877940462 us; error 68 us; skew 500 ppm
I20260925 15:07:57.941083 13881 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:57 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348877, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:57.945963 13881 init.cc:377] Logged in from keytab as oryx/127.4.205.194@KRBTEST.COM (short username oryx)
I20260925 15:07:57.947402 13881 webserver.cc:460] Webserver started at http://127.4.205.194:37367/ using document root <none> and password file <none>
I20260925 15:07:57.948004 13881 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:57.948127 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:57.953161 13881 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:07:57.956043 13897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.957649 13881 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 15:07:57.957799 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "9dbfae420a1744d3b02e4c135f4bb171"
format_stamp: "Formatted at 2026-09-25 15:07:55 on dist-test-slave-64fx"
server_key: "93edb4ef7cae4f36807da47f6a66ee44"
server_key_iv: "5f020d4ef5eaf3c020773b202c4dc421"
server_key_version: "encryptionkey@0"
I20260925 15:07:57.958747 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:57.975021 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:57.979061 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:57.979305 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:57.980252 13881 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:57.981642 13881 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:57.981762 13881 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:57.981887 13881 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:57.981980 13881 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:58.021790 14009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:36953 every 8 connection(s)
I20260925 15:07:58.021797 13881 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:36953
I20260925 15:07:58.024355 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:07:58.025382 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 13881
I20260925 15:07:58.026150 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:35885
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:42671
--webserver_port=38575
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:34011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:58.600742 14014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:07:58.601037 14014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:07:58.601191 14014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:07:58.607900 14014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:07:58.608033 14014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:07:58.608211 14014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:07:58.617436 14014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/krb5kdc/oryx_127.4.205.195.keytab
--principal=oryx/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:35885
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=38575
--tserver_master_addrs=127.4.205.254:42671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:07:58.618610 14014 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:07:58.619804 14014 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:07:58.636015 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:58.636037 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:07:58.644766 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:07:58.645216 14014 server_base.cc:1034] running on GCE node
I20260925 15:07:58.645937 14014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:07:58.646960 14014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:07:58.648164 14014 hybrid_clock.cc:648] HybridClock initialized: now 1790348878648103 us; error 63 us; skew 500 ppm
I20260925 15:07:58.648475 14014 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348878, etypes {rep=17 tkt=17 ses=17}, oryx/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:07:58.651610 14014 init.cc:377] Logged in from keytab as oryx/127.4.205.195@KRBTEST.COM (short username oryx)
I20260925 15:07:58.653384 14014 webserver.cc:460] Webserver started at http://127.4.205.195:38575/ using document root <none> and password file <none>
I20260925 15:07:58.654114 14014 fs_manager.cc:359] Metadata directory not provided
I20260925 15:07:58.654250 14014 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:07:58.660209 14014 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:07:58.663187 14030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:58.664634 14014 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:07:58.664801 14014 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "638a244ed56f4c3a908c985b1a48cdaf"
format_stamp: "Formatted at 2026-09-25 15:07:56 on dist-test-slave-64fx"
server_key: "421851a44a1ef3edc881eac99622a867"
server_key_iv: "6050018625a6c46f95938f31e0eff142"
server_key_version: "encryptionkey@0"
I20260925 15:07:58.665745 14014 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:07:58.703218 14014 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:07:58.707350 14014 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:07:58.707613 14014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:07:58.708724 14014 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:07:58.710132 14014 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:07:58.710246 14014 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:58.710364 14014 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:07:58.710435 14014 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:07:58.751741 14014 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:35885
I20260925 15:07:58.751783 14142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:35885 every 8 connection(s)
I20260925 15:07:58.753111 14014 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:07:58.759086 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14014
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:58.789014 13880 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.219s user 0.001s sys 0.011s
I20260925 15:07:58.790947 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:57.560079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:57821)
0925 15:07:57.560830 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:57.560870 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:57.562668 (+ 1798us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:57.563826 (+ 1158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:57.563833 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:57.564291 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:57.564851 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:57.564867 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:57.566574 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:57.566580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:57.567947 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:57.567959 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:57.568874 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:57.568881 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:57.569137 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:57.569783 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:57.569804 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:58.789837 (+1220033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":330,"threads_started":1}
W20260925 15:07:58.791528 13879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:57.559963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57821 (local address 127.4.205.254:42671)
0925 15:07:57.560711 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:57.560722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:57.560977 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:07:57.563085 (+ 2108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:57.563097 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:57.563393 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:57.563692 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:57.565032 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:57.566411 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:57.568069 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:57.568662 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:57.568852 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.790317 (+1221465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:57821: BlockingRecv error: recv got EOF from 127.4.205.193:57821 (error 108)
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":349,"threads_started":1}
W20260925 15:07:58.791872 13877 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:42671 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.205.254:42671: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
I20260925 15:07:58.800510 14146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:58.760549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49460 (local address 127.4.205.254:42671)
0925 15:07:58.770237 (+ 9688us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:58.770248 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:58.770275 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:07:58.770373 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:58.770381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:58.770487 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:58.770652 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:58.774899 (+ 4247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.776861 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.779752 (+ 2891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.780738 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.780890 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.785971 (+ 5081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:07:58.785985 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:07:58.786012 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:07:58.786068 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:07:58.794830 (+ 8762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:58.796014 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:58.796027 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:58.796039 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:58.796303 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:07:58.796852 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:07:58.796859 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:07:58.796865 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:07:58.797705 (+ 840us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:07:58.798109 (+ 404us) server_negotiation.cc:1035] Waiting for connection context
0925 15:07:58.799740 (+ 1631us) server_negotiation.cc:299] Negotiation successful
0925 15:07:58.800063 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":9503,"thread_start_us":229,"threads_started":1}
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:58.803797 13879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:58.792686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:52201 (local address 127.4.205.254:42671)
0925 15:07:58.792864 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:58.792873 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:58.792897 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:07:58.792956 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:58.792963 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:58.793047 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:58.793199 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:58.794044 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.795213 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.797186 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.797961 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.798089 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.803569 (+ 5480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:52201: BlockingRecv error: recv got EOF from 127.4.205.193:52201 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":38}
I20260925 15:07:58.804353 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:58.792466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:52201)
0925 15:07:58.792621 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:58.792637 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:58.792726 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:58.793224 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:58.793228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:58.793646 (+ 418us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:58.793841 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.793852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.795375 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.795383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.797020 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.797039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.798305 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.798313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.798528 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.799170 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:58.799194 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:58.803970 (+ 4776us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:58 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:07:58.819423 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:58.804958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:44147)
0925 15:07:58.805079 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:58.805103 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:58.805224 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:58.805976 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:58.805983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:58.806526 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:58.806811 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.806826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.808682 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.808690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.810852 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.810872 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.812305 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.812313 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.812587 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.813305 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:58.813334 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:58.819058 (+ 5724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 15:07:58.819783 13877 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:07:58.819927 13879 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:58.805105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:44147 (local address 127.4.205.254:42671)
0925 15:07:58.805600 (+ 495us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:58.805609 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:58.805630 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:07:58.805687 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:58.805693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:58.805793 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:58.805916 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:58.806993 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.808511 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.811070 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.812115 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.812276 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.819304 (+ 7028us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:44147: BlockingRecv error: recv got EOF from 127.4.205.193:44147 (error 108)
Metrics: {"server-negotiator.queue_time_us":313}
W20260925 15:07:59.288343 14012 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.220s user 0.008s sys 0.018s
W20260925 15:07:59.290208 14013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:58.028520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:56391 (local address 127.4.205.254:42671)
0925 15:07:58.050161 (+ 21641us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:58.050172 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:58.051414 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 15:07:58.055203 (+ 3789us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:58.055210 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:58.055298 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:58.055418 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:58.058608 (+ 3190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.059966 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.063982 (+ 4016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.065148 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.065318 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.289899 (+1224581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:56391: BlockingRecv error: recv got EOF from 127.4.205.194:56391 (error 108)
Metrics: {"server-negotiator.queue_time_us":21447,"thread_start_us":21293,"threads_started":1}
I20260925 15:07:59.291189 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:58.028003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:56391)
0925 15:07:58.051009 (+ 23006us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:58.051085 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:58.054723 (+ 3638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:58.055831 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:58.055850 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:58.056974 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:58.058333 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.058408 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.060154 (+ 1746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.060163 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.063700 (+ 3537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.063720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.066291 (+ 2571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.066301 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.066747 (+ 446us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.067958 (+ 1211us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:58.068098 (+ 140us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.289519 (+1221421us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":22522,"thread_start_us":22281,"threads_started":1}
W20260925 15:07:59.292095 14010 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:42671 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.205.254:42671: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:07:59.302244 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:59.292718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:50901)
0925 15:07:59.292895 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:59.292915 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:59.293021 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:59.293323 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:59.293335 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:59.293796 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:59.293942 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.293949 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.294947 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.294953 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.295868 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.295878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.296567 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.296572 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.296743 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.297469 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:59.297485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.301868 (+ 4383us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:07:59.302668 14013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:59.292821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:50901 (local address 127.4.205.254:42671)
0925 15:07:59.292985 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:59.292990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:59.293006 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:07:59.293120 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:59.293125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:59.293207 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:59.293296 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:59.294065 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.294796 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.296017 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.296411 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.296539 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.302476 (+ 5937us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:50901: BlockingRecv error: recv got EOF from 127.4.205.194:50901 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:59.314524 14013 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:59.303545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44115 (local address 127.4.205.254:42671)
0925 15:07:59.303695 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:59.303698 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:59.303712 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:07:59.303844 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:59.303847 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:59.303902 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:59.303973 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:59.305008 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.305665 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.308019 (+ 2354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.308348 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.308427 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.314363 (+ 5936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:44115: BlockingRecv error: recv got EOF from 127.4.205.194:44115 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:07:59.314606 14012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:59.303362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:44115)
0925 15:07:59.303560 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:59.303586 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:59.303715 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:59.304051 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:59.304059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:59.304581 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:59.304846 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.304861 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.305798 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.305806 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.307839 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.307857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.308495 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.308504 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.308758 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.309396 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:59.309424 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.314247 (+ 4823us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:07:59.314996 14010 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 15:07:59.831740 14151 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:59.820424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:51537 (local address 127.4.205.254:42671)
0925 15:07:59.822095 (+ 1671us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:59.822100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:59.822114 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:07:59.822161 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:59.822164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:59.822215 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:59.822298 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:59.823344 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.824031 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.825991 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.826387 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.826474 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.831562 (+ 5088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:51537: BlockingRecv error: recv got EOF from 127.4.205.193:51537 (error 108)
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":1477,"threads_started":1}
I20260925 15:07:59.832319 14150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:59.820363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:51537)
0925 15:07:59.821841 (+ 1478us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:59.821862 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:59.821982 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:59.822353 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:59.822358 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:59.822969 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:59.823183 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.823195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.824166 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.824171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.825850 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.825869 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.826556 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.826563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.826747 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.827338 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:59.827361 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.832019 (+ 4658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1337,"thread_start_us":1214,"threads_started":1}
W20260925 15:07:59.858093 14145 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.076s user 0.006s sys 0.010s
W20260925 15:07:59.859127 14148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:58.757662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:52983 (local address 127.4.205.254:42671)
0925 15:07:58.761053 (+ 3391us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:58.761061 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:58.770632 (+ 9571us) server_negotiation.cc:407] Connection header received
0925 15:07:58.773339 (+ 2707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:58.773344 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:58.773426 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:58.773490 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:58.775720 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.776977 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.779342 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.779983 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.780086 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.858887 (+1078801us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:52983: BlockingRecv error: recv got EOF from 127.4.205.195:52983 (error 108)
Metrics: {"server-negotiator.queue_time_us":3228,"thread_start_us":12246,"threads_started":1}
I20260925 15:07:59.859642 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:58.756925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:52983)
0925 15:07:58.770487 (+ 13562us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:58.770529 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:58.773015 (+ 2486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:58.774082 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:58.774093 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:58.774771 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:58.775525 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.775546 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.777161 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.777166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.779197 (+ 2031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:58.779210 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:58.780461 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:58.780467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:58.780730 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:58.781969 (+ 1239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:58.781996 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.858746 (+1076750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":13255,"thread_start_us":16672,"threads_started":1}
W20260925 15:07:59.860535 14143 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:42671 (0 consecutive failures): Not authorized: Failed to ping master at 127.4.205.254:42671: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:07:59.868953 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:59.861112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:59843)
0925 15:07:59.861285 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:59.861327 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:59.861424 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:59.861746 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:59.861749 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:59.862044 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:59.862236 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.862244 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.863239 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.863243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.864130 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.864140 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.864723 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.864727 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.864824 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.865418 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:59.865433 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.868756 (+ 3323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:07:59.869112 14148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:59.861152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:59843 (local address 127.4.205.254:42671)
0925 15:07:59.861388 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:59.861392 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:59.861409 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:59.861518 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:59.861521 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:59.861619 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:59.861710 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:59.862358 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.863090 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.864247 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.864600 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.864679 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.868926 (+ 4247us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:59843: BlockingRecv error: recv got EOF from 127.4.205.195:59843 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:07:59 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:07:59.876657 14145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:07:59.869598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:38319)
0925 15:07:59.869743 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:07:59.869761 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:07:59.869878 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:07:59.870177 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:07:59.870184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:07:59.870556 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:07:59.870689 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.870697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.871593 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.871600 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.872508 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:07:59.872517 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.873187 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.873193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:07:59.873387 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.873870 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:07:59.873884 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:07:59.876417 (+ 2533us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":32}
W20260925 15:07:59.876798 14148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:07:59.869639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:38319 (local address 127.4.205.254:42671)
0925 15:07:59.869799 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:07:59.869822 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 15:07:59.869839 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:07:59.869982 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:07:59.869987 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:07:59.870051 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:07:59.870150 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:07:59.870815 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.871475 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.872635 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:07:59.873044 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:07:59.873168 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:07:59.876571 (+ 3403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:38319: BlockingRecv error: recv got EOF from 127.4.205.195:38319 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260925 15:07:59.876919 14143 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:00.329053 14152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:00.315563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:56217)
0925 15:08:00.316073 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:00.316088 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:00.316188 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:00.316494 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:00.316499 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:00.317114 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:00.317333 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.317348 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.318381 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.318384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.319278 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.319288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.319945 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.319952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.320111 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.325350 (+ 5239us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:00.325372 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:00.328826 (+ 3454us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":291,"threads_started":1}
W20260925 15:08:00.329205 14148 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:00.315609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:56217 (local address 127.4.205.254:42671)
0925 15:08:00.315749 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:00.315753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:00.316153 (+ 400us) server_negotiation.cc:407] Connection header received
0925 15:08:00.316272 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:00.316275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:00.316339 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:00.316415 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:00.317482 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.318260 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.319390 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.319769 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.319877 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.329010 (+ 9133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:56217: BlockingRecv error: recv got EOF from 127.4.205.194:56217 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:00.842160 14153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:00.833325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:33469)
0925 15:08:00.833936 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:00.833970 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:00.834097 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:00.834671 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:00.834674 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:00.835098 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:00.835274 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.835281 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.836212 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.836215 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.837101 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.837110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.837841 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.837845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.837964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.838579 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:00.838595 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:00.841928 (+ 3333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":361,"threads_started":1}
W20260925 15:08:00.842555 14154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:00.833385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:33469 (local address 127.4.205.254:42671)
0925 15:08:00.834364 (+ 979us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:00.834370 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:00.834386 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:08:00.834437 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:00.834443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:00.834534 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:00.834648 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:00.835393 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.836106 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.837223 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.837720 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.837812 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.842133 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:33469: BlockingRecv error: recv got EOF from 127.4.205.193:33469 (error 108)
Metrics: {"server-negotiator.queue_time_us":863,"thread_start_us":776,"threads_started":1}
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:00 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:00.886446 14155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:00.877562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:57165)
0925 15:08:00.878111 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:00.878126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:00.878230 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:00.878514 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:00.878517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:00.878892 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:00.879034 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.879041 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.879995 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.880000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.881546 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:00.881600 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.882223 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.882227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:00.882334 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.882946 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:00.882964 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:00.886194 (+ 3230us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":295,"threads_started":1}
W20260925 15:08:00.886617 14154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:00.877651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:57165 (local address 127.4.205.254:42671)
0925 15:08:00.877798 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:00.877802 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:00.878194 (+ 392us) server_negotiation.cc:407] Connection header received
0925 15:08:00.878319 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:00.878323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:00.878392 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:00.878468 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:00.879146 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.879823 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.881708 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:00.882089 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:00.882177 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:00.886419 (+ 4242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:57165: BlockingRecv error: recv got EOF from 127.4.205.195:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:01.339013 14156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:01.330103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:56825)
0925 15:08:01.330580 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:01.330601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:01.330729 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:01.331137 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:01.331141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:01.331556 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:01.331725 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.331733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.332730 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.332737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.333696 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.333707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.334428 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.334433 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.334616 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.335241 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:01.335258 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:01.338668 (+ 3410us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":235,"threads_started":1}
W20260925 15:08:01.339078 14154 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:01.330431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:56825 (local address 127.4.205.254:42671)
0925 15:08:01.330623 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:01.330631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:01.330655 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:08:01.330895 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:01.330902 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:01.330983 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:01.331086 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:01.331872 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.332591 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.333844 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.334261 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.334415 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.338836 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:56825: BlockingRecv error: recv got EOF from 127.4.205.194:56825 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:01.851104 14157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:01.842929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:50663)
0925 15:08:01.843419 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:01.843435 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:01.843569 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:01.843863 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:01.843866 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:01.844180 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:01.844331 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.844338 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.845323 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.845327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.846264 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.846274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.846883 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.846888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.847062 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.847583 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:01.847599 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:01.850882 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":240,"threads_started":1}
W20260925 15:08:01.851311 14158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:01.843085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50663 (local address 127.4.205.254:42671)
0925 15:08:01.843533 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:01.843537 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:01.843553 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:08:01.843667 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:01.843671 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:01.843742 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:01.843827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:01.844468 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.845206 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.846390 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.846756 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.846841 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.851056 (+ 4215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:50663: BlockingRecv error: recv got EOF from 127.4.205.193:50663 (error 108)
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":256,"threads_started":1}
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:01 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:01.895176 14159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:01.887439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:40965)
0925 15:08:01.887913 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:01.887931 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:01.888049 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:01.888314 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:01.888317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:01.888662 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:01.888828 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.888836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.889741 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.889744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.890651 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:01.890661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.891285 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.891288 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:01.891402 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.891899 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:01.891914 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:01.894960 (+ 3046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":270,"threads_started":1}
W20260925 15:08:01.895304 14158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:01.887749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:40965 (local address 127.4.205.254:42671)
0925 15:08:01.887912 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:01.887920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:01.888012 (+ 92us) server_negotiation.cc:407] Connection header received
0925 15:08:01.888141 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:01.888144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:01.888201 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:01.888276 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:01.888950 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.889639 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.890765 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:01.891153 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:01.891254 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:01.895120 (+ 3866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:40965: BlockingRecv error: recv got EOF from 127.4.205.195:40965 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:02.348664 14160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:02.339932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:44261)
0925 15:08:02.340473 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:02.340496 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:02.340600 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:02.340875 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:02.340878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:02.341258 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:02.341489 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.341499 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.342727 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.342735 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.343663 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.343673 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.344356 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.344360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.344538 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.345130 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:02.345146 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:02.348434 (+ 3288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":299,"threads_started":1}
W20260925 15:08:02.348875 14158 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:02.340008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44261 (local address 127.4.205.254:42671)
0925 15:08:02.340187 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:02.340191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:02.340564 (+ 373us) server_negotiation.cc:407] Connection header received
0925 15:08:02.340688 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:02.340692 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:02.340760 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:02.340838 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:02.341656 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.342601 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.343803 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.344218 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.344347 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.348617 (+ 4270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:44261: BlockingRecv error: recv got EOF from 127.4.205.194:44261 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:02.860234 14161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:02.851948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:40209)
0925 15:08:02.852358 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:02.852373 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:02.852473 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:02.852928 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:02.852935 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:02.853333 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:02.853487 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.853494 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.854746 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.854749 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.855692 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.855703 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.856240 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.856243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.856350 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.856827 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:02.856842 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:02.859983 (+ 3141us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":211,"threads_started":1}
W20260925 15:08:02.860424 14162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:02.852019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:40209 (local address 127.4.205.254:42671)
0925 15:08:02.852642 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:02.852646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:02.852659 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:08:02.852701 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:02.852704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:02.852758 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:02.852835 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:02.853649 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.854638 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.855807 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.856141 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.856219 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.860182 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:40209: BlockingRecv error: recv got EOF from 127.4.205.193:40209 (error 108)
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":435,"threads_started":1}
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:02 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:02.904307 14163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:02.895901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:45863)
0925 15:08:02.896383 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:02.896398 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:02.896498 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:02.896757 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:02.896759 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:02.897075 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:02.897221 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.897229 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.898147 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.898150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.899124 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:02.899136 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.900054 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.900059 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:02.900262 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.900816 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:02.900832 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:02.903976 (+ 3144us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":262,"threads_started":1}
W20260925 15:08:02.904351 14162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:02.896009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:45863 (local address 127.4.205.254:42671)
0925 15:08:02.896211 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:02.896215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:02.896467 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:08:02.896586 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:02.896590 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:02.896642 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:02.896716 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:02.897337 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.898029 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.899246 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:02.899892 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:02.900011 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:02.904142 (+ 4131us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:45863: BlockingRecv error: recv got EOF from 127.4.205.195:45863 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:03.358366 14164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:03.349620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:58303)
0925 15:08:03.350194 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:03.350209 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:03.350358 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:03.350983 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:03.350992 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:03.351415 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:03.351569 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.351578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.352517 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.352520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.353467 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.353477 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.354147 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.354151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.354264 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.354905 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:03.354921 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:03.358010 (+ 3089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":320,"threads_started":1}
W20260925 15:08:03.358433 14162 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:03.349690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:58303 (local address 127.4.205.254:42671)
0925 15:08:03.349894 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:03.349901 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:03.350320 (+ 419us) server_negotiation.cc:407] Connection header received
0925 15:08:03.350707 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:03.350712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:03.350815 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:03.350907 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:03.351689 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.352410 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.353614 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.354038 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.354126 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.358222 (+ 4096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:58303: BlockingRecv error: recv got EOF from 127.4.205.194:58303 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:03.869398 14165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:03.861072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:54133)
0925 15:08:03.861535 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:03.861592 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:03.861729 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:03.862150 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:03.862158 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:03.862580 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:03.862722 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.862730 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.863644 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.863647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.864641 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.864652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.865307 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.865312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.865490 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.866083 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:03.866110 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:03.869192 (+ 3082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":248,"threads_started":1}
W20260925 15:08:03.869643 14166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:03.861148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:54133 (local address 127.4.205.254:42671)
0925 15:08:03.861819 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:03.861823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:03.861837 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:08:03.861892 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:03.861895 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:03.861955 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:03.862037 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:03.862836 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.863546 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.864784 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.865162 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.865288 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.869378 (+ 4090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:54133: BlockingRecv error: recv got EOF from 127.4.205.193:54133 (error 108)
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":467,"threads_started":1}
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:03 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:03.913715 14167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:03.905248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:59919)
0925 15:08:03.905730 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:03.905753 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:03.905846 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:03.906106 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:03.906108 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:03.906429 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:03.906570 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.906578 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.907466 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.907469 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.908403 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:03.908412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.908966 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.908969 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:03.909078 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.909644 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:03.909659 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:03.913344 (+ 3685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":252,"threads_started":1}
W20260925 15:08:03.913748 14166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:03.905296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:59919 (local address 127.4.205.254:42671)
0925 15:08:03.905430 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:03.905434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:03.905815 (+ 381us) server_negotiation.cc:407] Connection header received
0925 15:08:03.905929 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:03.905932 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:03.905991 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:03.906067 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:03.906676 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.907354 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.908506 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:03.908867 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:03.908944 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:03.913507 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:59919: BlockingRecv error: recv got EOF from 127.4.205.195:59919 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:04.368031 14168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:04.359246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:44741)
0925 15:08:04.359790 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:04.359806 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:04.359913 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:04.360204 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:04.360207 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:04.360621 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:04.360769 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.360777 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.361918 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.361926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.362974 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.362985 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.363698 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.363703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.363874 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.364461 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:04.364478 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:04.367822 (+ 3344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":301,"threads_started":1}
W20260925 15:08:04.368338 14166 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:04.359383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44741 (local address 127.4.205.254:42671)
0925 15:08:04.359544 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:04.359548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:04.359887 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:08:04.360009 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:04.360013 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:04.360083 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:04.360162 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:04.360895 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.361768 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.363119 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.363557 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.363683 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.368042 (+ 4359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:44741: BlockingRecv error: recv got EOF from 127.4.205.194:44741 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:04.879150 14169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:04.870219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:40439)
0925 15:08:04.870682 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:04.870706 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:04.870837 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:04.871171 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:04.871176 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:04.871595 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:04.871755 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.871764 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.872788 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.872795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.873729 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.873740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.874431 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.874436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.874614 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.875250 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:04.875269 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:04.878828 (+ 3559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":210,"threads_started":1}
W20260925 15:08:04.879247 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:04.870294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:40439 (local address 127.4.205.254:42671)
0925 15:08:04.870869 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:04.870877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:04.870898 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:08:04.870952 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:04.870963 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:04.871038 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:04.871142 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:04.871901 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.872654 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.873887 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.874285 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.874415 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.879021 (+ 4606us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:40439: BlockingRecv error: recv got EOF from 127.4.205.193:40439 (error 108)
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":380,"threads_started":1}
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:04 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:04.922953 14171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:04.914632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:52663)
0925 15:08:04.915091 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:04.915108 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:04.915214 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:04.915483 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:04.915486 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:04.915853 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:04.915995 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.916003 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.916939 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.916942 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.917960 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:04.917970 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.918587 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.918591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:04.918703 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.919303 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:04.919320 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:04.922644 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":258,"threads_started":1}
W20260925 15:08:04.923171 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:04.914668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:52663 (local address 127.4.205.254:42671)
0925 15:08:04.914875 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:04.914883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:04.915184 (+ 301us) server_negotiation.cc:407] Connection header received
0925 15:08:04.915305 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:04.915309 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:04.915367 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:04.915444 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:04.916105 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.916830 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.918071 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:04.918478 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:04.918563 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:04.922822 (+ 4259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:52663: BlockingRecv error: recv got EOF from 127.4.205.195:52663 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:05.383971 14172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:05.368906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:55027)
0925 15:08:05.369373 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:05.369389 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:05.369499 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:05.369829 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:05.369832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:05.370239 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:05.370418 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.370427 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.371445 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.371451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.372904 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.372915 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.373531 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.373535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.373777 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.379945 (+ 6168us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:05.379962 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:05.383612 (+ 3650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":278,"threads_started":1}
W20260925 15:08:05.384123 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:05.368927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:55027 (local address 127.4.205.254:42671)
0925 15:08:05.369141 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:05.369150 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:05.369475 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:08:05.369629 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:05.369641 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:05.369713 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:05.369791 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:05.370540 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.371284 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.373029 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.373410 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.373502 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.383878 (+ 10376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:55027: BlockingRecv error: recv got EOF from 127.4.205.194:55027 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:05.888664 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:05.880045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:32997)
0925 15:08:05.880540 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:05.880555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:05.880669 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:05.880937 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:05.880941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:05.881341 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:05.881502 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.881510 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.882471 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.882474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.883507 (+ 1033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.883518 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.884194 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.884198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.884393 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.884970 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:05.884987 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:05.888456 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":244,"threads_started":1}
W20260925 15:08:05.888804 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:05.880153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:32997 (local address 127.4.205.254:42671)
0925 15:08:05.880394 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:05.880400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:05.880634 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:08:05.880747 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:05.880751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:05.880809 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:05.880886 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:05.881679 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.882363 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.883645 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.884076 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.884164 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.888615 (+ 4451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:32997: BlockingRecv error: recv got EOF from 127.4.205.193:32997 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:05 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:05.932456 14174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:05.923843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:51461)
0925 15:08:05.924292 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:05.924308 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:05.924483 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:05.924821 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:05.924825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:05.925266 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:05.925426 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.925435 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.926467 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.926474 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.927374 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:05.927384 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.928157 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.928163 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:05.928345 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.928997 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:05.929013 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:05.932229 (+ 3216us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":237,"threads_started":1}
W20260925 15:08:05.932659 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:05.923898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:51461 (local address 127.4.205.254:42671)
0925 15:08:05.924110 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:05.924119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:05.924410 (+ 291us) server_negotiation.cc:407] Connection header received
0925 15:08:05.924603 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:05.924610 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:05.924679 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:05.924790 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:05.925596 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.926334 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.927510 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:05.927996 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:05.928142 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:05.932440 (+ 4298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:51461: BlockingRecv error: recv got EOF from 127.4.205.195:51461 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:06.393355 14175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:06.384962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:54079)
0925 15:08:06.385614 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:06.385630 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:06.385733 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:06.386017 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:06.386020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:06.386388 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:06.386547 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.386554 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.387462 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.387465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.388381 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.388391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.389145 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.389150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.389304 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.390007 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:06.390033 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:06.393132 (+ 3099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":314,"threads_started":1}
W20260925 15:08:06.393488 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:06.385015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:54079 (local address 127.4.205.254:42671)
0925 15:08:06.385226 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:06.385230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:06.385697 (+ 467us) server_negotiation.cc:407] Connection header received
0925 15:08:06.385826 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:06.385830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:06.385900 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:06.385983 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:06.386660 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.387348 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.388491 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.388999 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.389121 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.393297 (+ 4176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:54079: BlockingRecv error: recv got EOF from 127.4.205.194:54079 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:06.898728 14176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:06.889844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:46537)
0925 15:08:06.890324 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:06.890338 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:06.890511 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:06.890848 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:06.890852 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:06.891316 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:06.891515 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.891524 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.892670 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.892677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.893673 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.893684 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.894408 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.894413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.894593 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.895212 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:06.895229 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:06.898489 (+ 3260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
W20260925 15:08:06.898874 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:06.889934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:46537 (local address 127.4.205.254:42671)
0925 15:08:06.890147 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:06.890153 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:06.890426 (+ 273us) server_negotiation.cc:407] Connection header received
0925 15:08:06.890629 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:06.890635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:06.890701 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:06.890818 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:06.891670 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.892532 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.893809 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.894232 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.894376 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.898682 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:46537: BlockingRecv error: recv got EOF from 127.4.205.193:46537 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:06 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:06.942253 14177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:06.933352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:54181)
0925 15:08:06.934036 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:06.934052 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:06.934203 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:06.934573 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:06.934579 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:06.934993 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:06.935152 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.935160 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.936117 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.936123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.937037 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:06.937047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.937772 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.937777 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:06.937988 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.938654 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:06.938671 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:06.942012 (+ 3341us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":373,"threads_started":1}
W20260925 15:08:06.942505 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:06.933685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:54181 (local address 127.4.205.254:42671)
0925 15:08:06.933904 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:06.933910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:06.934143 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:08:06.934334 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:06.934343 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:06.934409 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:06.934527 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:06.935297 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.935976 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.937174 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:06.937632 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:06.937757 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:06.942244 (+ 4487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:54181: BlockingRecv error: recv got EOF from 127.4.205.195:54181 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:07.403048 14178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:07.394397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:59433)
0925 15:08:07.394884 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:07.394902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:07.395031 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:07.395416 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:07.395421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:07.395937 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:07.396100 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.396134 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.397295 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.397299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.398196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.398205 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.398844 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.398849 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.399059 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.399620 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:07.399636 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:07.402817 (+ 3181us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":255,"threads_started":1}
W20260925 15:08:07.403177 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:07.394451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:59433 (local address 127.4.205.254:42671)
0925 15:08:07.394657 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:07.394667 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:07.394992 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:08:07.395172 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:07.395177 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:07.395266 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:07.395359 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:07.396278 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.397175 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.398309 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.398686 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.398770 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.402981 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:59433: BlockingRecv error: recv got EOF from 127.4.205.194:59433 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:07.908221 14179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:07.899691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:50479)
0925 15:08:07.900183 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:07.900199 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:07.900306 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:07.900606 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:07.900609 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:07.900982 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:07.901116 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.901123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.902295 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.902299 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.903196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.903206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.904218 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.904222 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.904333 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.904863 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:07.904896 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:07.908008 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
W20260925 15:08:07.908378 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:07.899742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50479 (local address 127.4.205.254:42671)
0925 15:08:07.899904 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:07.899908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:07.900267 (+ 359us) server_negotiation.cc:407] Connection header received
0925 15:08:07.900420 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:07.900424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:07.900482 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:07.900566 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:07.901242 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.902183 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.903356 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.904101 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.904193 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.908182 (+ 3989us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:50479: BlockingRecv error: recv got EOF from 127.4.205.193:50479 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:07 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:07.951488 14180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:07.943091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:44729)
0925 15:08:07.943614 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:07.943629 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:07.943775 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:07.944098 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:07.944102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:07.944478 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:07.944624 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.944631 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.945644 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.945650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.946596 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:07.946606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.947308 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.947314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:07.947483 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.948021 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:07.948037 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:07.951149 (+ 3112us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":230,"threads_started":1}
W20260925 15:08:07.951572 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:07.943237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:44729 (local address 127.4.205.254:42671)
0925 15:08:07.943455 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:07.943462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:07.943714 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:08:07.943891 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:07.943898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:07.943972 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:07.944072 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:07.944757 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.945473 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.946728 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:07.947143 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:07.947276 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:07.951330 (+ 4054us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:44729: BlockingRecv error: recv got EOF from 127.4.205.195:44729 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:08.412925 14181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:08.404183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:54881)
0925 15:08:08.404682 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:08.404700 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:08.404829 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:08.405212 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:08.405219 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:08.405763 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:08.405951 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.405965 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.406967 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.406973 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.407874 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.407885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.408586 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.408590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.408715 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.409310 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:08.409326 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:08.412708 (+ 3382us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":322,"threads_started":1}
W20260925 15:08:08.413089 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:08.404208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:54881 (local address 127.4.205.254:42671)
0925 15:08:08.404403 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:08.404407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:08.404766 (+ 359us) server_negotiation.cc:407] Connection header received
0925 15:08:08.404948 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:08.404953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:08.405054 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:08.405165 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:08.406097 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.406836 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.408011 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.408427 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.408569 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.412871 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:54881: BlockingRecv error: recv got EOF from 127.4.205.194:54881 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:08.917290 14182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:08.909035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:37145)
0925 15:08:08.909477 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:08.909492 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:08.909662 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:08.909961 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:08.909964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:08.910312 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:08.910473 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.910480 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.911588 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.911592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.912688 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.912699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.913316 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.913320 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.913431 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.914073 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:08.914089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:08.917060 (+ 2971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":236,"threads_started":1}
W20260925 15:08:08.917501 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:08.909119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37145 (local address 127.4.205.254:42671)
0925 15:08:08.909334 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:08.909340 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:08.909619 (+ 279us) server_negotiation.cc:407] Connection header received
0925 15:08:08.909786 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:08.909790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:08.909847 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:08.909919 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:08.910592 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.911481 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.912800 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.913187 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.913324 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.917231 (+ 3907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:37145: BlockingRecv error: recv got EOF from 127.4.205.193:37145 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:08 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:08.961192 14183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:08.952539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:43027)
0925 15:08:08.953064 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:08.953080 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:08.953207 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:08.953653 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:08.953658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:08.954246 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:08.954444 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.954453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.955357 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.955360 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.956236 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:08.956245 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.956922 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.956926 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:08.957047 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.957591 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:08.957607 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:08.960964 (+ 3357us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":293,"threads_started":1}
W20260925 15:08:08.961337 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:08.952549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:43027 (local address 127.4.205.254:42671)
0925 15:08:08.952739 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:08.952745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:08.953158 (+ 413us) server_negotiation.cc:407] Connection header received
0925 15:08:08.953331 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:08.953336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:08.953449 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:08.953526 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:08.954571 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.955248 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.956346 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:08.956804 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:08.956892 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:08.961124 (+ 4232us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:43027: BlockingRecv error: recv got EOF from 127.4.205.195:43027 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:09.422252 14184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:09.413818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:48939)
0925 15:08:09.414336 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:09.414353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:09.414462 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:09.414751 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:09.414754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:09.415099 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:09.415251 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.415259 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.416161 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.416164 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.417139 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.417149 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.417828 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.417833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.417945 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.418524 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:09.418539 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:09.422034 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":298,"threads_started":1}
W20260925 15:08:09.422401 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:09.413900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:48939 (local address 127.4.205.254:42671)
0925 15:08:09.414074 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:09.414078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:09.414423 (+ 345us) server_negotiation.cc:407] Connection header received
0925 15:08:09.414559 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:09.414563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:09.414632 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:09.414706 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:09.415367 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.416050 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.417250 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.417674 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.417759 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.422194 (+ 4435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:48939: BlockingRecv error: recv got EOF from 127.4.205.194:48939 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:09.926699 14185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:09.918329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:51743)
0925 15:08:09.918883 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:09.918898 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:09.919033 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:09.919437 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:09.919445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:09.919883 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:09.920079 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.920089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.921010 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.921013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.921986 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.921997 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.922649 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.922657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.922829 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.923379 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:09.923406 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:09.926478 (+ 3072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":341,"threads_started":1}
W20260925 15:08:09.926867 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:09.918413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:51743 (local address 127.4.205.254:42671)
0925 15:08:09.918643 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:09.918649 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:09.918959 (+ 310us) server_negotiation.cc:407] Connection header received
0925 15:08:09.919167 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:09.919172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:09.919244 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:09.919389 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:09.920196 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.920896 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.922103 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.922501 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.922583 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.926643 (+ 4060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:51743: BlockingRecv error: recv got EOF from 127.4.205.193:51743 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:09 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:09.970712 14186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:09.962160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:37533)
0925 15:08:09.962632 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:09.962653 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:09.962765 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:09.963019 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:09.963022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:09.963374 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:09.963540 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.963547 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.964387 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.964390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.965522 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:09.965539 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.966472 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.966476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:09.966582 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.967115 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:09.967130 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:09.970368 (+ 3238us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":237,"threads_started":1}
W20260925 15:08:09.970772 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:09.962210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:37533 (local address 127.4.205.254:42671)
0925 15:08:09.962432 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:09.962451 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:09.962734 (+ 283us) server_negotiation.cc:407] Connection header received
0925 15:08:09.962848 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:09.962851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:09.962905 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:09.962999 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:09.963644 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.964287 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.965693 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:09.966342 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:09.966443 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:09.970538 (+ 4095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:37533: BlockingRecv error: recv got EOF from 127.4.205.195:37533 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:10.436493 14187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:10.423103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:54703)
0925 15:08:10.423507 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:10.423522 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:10.423627 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:10.423880 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:10.423883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:10.424257 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:10.424416 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.424424 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.425310 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.425313 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.426235 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.426246 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.426842 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.426846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.426965 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.432951 (+ 5986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:10.432968 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:10.436175 (+ 3207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":235,"threads_started":1}
W20260925 15:08:10.436582 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:10.423146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:54703 (local address 127.4.205.254:42671)
0925 15:08:10.423373 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:10.423379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:10.423593 (+ 214us) server_negotiation.cc:407] Connection header received
0925 15:08:10.423713 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:10.423717 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:10.423773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:10.423846 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:10.424524 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.425212 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.426344 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.426729 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.426812 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.436345 (+ 9533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:54703: BlockingRecv error: recv got EOF from 127.4.205.194:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:10.935767 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:10.927633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:35711)
0925 15:08:10.928166 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:10.928181 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:10.928287 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:10.928572 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:10.928575 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:10.928959 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:10.929137 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.929145 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.930079 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.930082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.930992 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.931002 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.931645 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.931649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.931764 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.932309 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:10.932341 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:10.935452 (+ 3111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":277,"threads_started":1}
W20260925 15:08:10.935870 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:10.927706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:35711 (local address 127.4.205.254:42671)
0925 15:08:10.927868 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:10.927872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:10.928250 (+ 378us) server_negotiation.cc:407] Connection header received
0925 15:08:10.928388 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:10.928391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:10.928446 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:10.928522 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:10.929250 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.929958 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.931104 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.931509 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.931594 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.935630 (+ 4036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:35711: BlockingRecv error: recv got EOF from 127.4.205.193:35711 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:10 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:10.980314 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:10.971704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:37403)
0925 15:08:10.972180 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:10.972199 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:10.972362 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:10.972673 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:10.972677 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:10.973070 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:10.973228 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.973235 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.974253 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.974278 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.975169 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:10.975179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.975853 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.975859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:10.976066 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.976758 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:10.976775 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:10.979972 (+ 3197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":240,"threads_started":1}
W20260925 15:08:10.980371 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:10.971755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:37403 (local address 127.4.205.254:42671)
0925 15:08:10.971925 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:10.971933 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:10.972287 (+ 354us) server_negotiation.cc:407] Connection header received
0925 15:08:10.972480 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:10.972487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:10.972552 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:10.972643 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:10.973363 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.974136 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.975306 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:10.975717 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:10.975838 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:10.980159 (+ 4321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:37403: BlockingRecv error: recv got EOF from 127.4.205.195:37403 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:11.446151 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:11.437307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:60015)
0925 15:08:11.437861 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:11.437882 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:11.438003 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:11.438266 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:11.438269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:11.438680 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:11.438837 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.438845 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.439809 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.439813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.441232 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.441246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.441993 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.441998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.442119 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.442688 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:11.442703 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:11.445921 (+ 3218us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":308,"threads_started":1}
W20260925 15:08:11.446359 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:11.437381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:60015 (local address 127.4.205.254:42671)
0925 15:08:11.437661 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:11.437667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:11.437963 (+ 296us) server_negotiation.cc:407] Connection header received
0925 15:08:11.438083 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:11.438087 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:11.438146 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:11.438220 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:11.438963 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.439678 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.441379 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.441852 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.441944 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.446112 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:60015: BlockingRecv error: recv got EOF from 127.4.205.194:60015 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:11.944985 14191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:11.936645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:39137)
0925 15:08:11.937076 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:11.937091 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:11.937196 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:11.937461 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:11.937464 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:11.937876 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:11.938015 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.938023 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.938899 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.938902 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.940005 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.940016 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.940854 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.940858 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.940994 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.941547 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:11.941616 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:11.944763 (+ 3147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":247,"threads_started":1}
W20260925 15:08:11.945132 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:11.936738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:39137 (local address 127.4.205.254:42671)
0925 15:08:11.936991 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:11.936996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:11.937164 (+ 168us) server_negotiation.cc:407] Connection header received
0925 15:08:11.937285 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:11.937288 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:11.937345 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:11.937423 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:11.938126 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.938791 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.940158 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.940738 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.940827 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.944929 (+ 4102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:39137: BlockingRecv error: recv got EOF from 127.4.205.193:39137 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:11 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:11.989930 14192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:11.981134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:51401)
0925 15:08:11.981701 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:11.981717 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:11.981813 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:11.982059 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:11.982062 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:11.982464 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:11.982630 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.982638 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.983651 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.983654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.984519 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:11.984529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.985202 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.985210 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:11.985341 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.986118 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:11.986143 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:11.989712 (+ 3569us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
W20260925 15:08:11.990077 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:11.981210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:51401 (local address 127.4.205.254:42671)
0925 15:08:11.981380 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:11.981384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:11.981764 (+ 380us) server_negotiation.cc:407] Connection header received
0925 15:08:11.981887 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:11.981890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:11.981944 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:11.982022 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:11.982738 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.983530 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.984625 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:11.985040 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:11.985122 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:11.989851 (+ 4729us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:51401: BlockingRecv error: recv got EOF from 127.4.205.195:51401 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:12.454999 14193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:12.447014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:51875)
0925 15:08:12.447459 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:12.447480 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:12.447584 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:12.447854 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:12.447857 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:12.448222 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:12.448360 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.448368 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.449248 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.449251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.450210 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.450223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.450825 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.450829 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.450940 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.451503 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:12.451519 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:12.454763 (+ 3244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":228,"threads_started":1}
W20260925 15:08:12.455204 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:12.447110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:51875 (local address 127.4.205.254:42671)
0925 15:08:12.447268 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:12.447272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:12.447546 (+ 274us) server_negotiation.cc:407] Connection header received
0925 15:08:12.447670 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:12.447673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:12.447741 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:12.447816 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:12.448470 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.449134 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.450326 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.450716 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.450800 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.454971 (+ 4171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:51875: BlockingRecv error: recv got EOF from 127.4.205.194:51875 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.193@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:12.954154 14194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:12.946182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.193:38065)
0925 15:08:12.946698 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:12.946713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:12.946813 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:12.947072 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:12.947076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:12.947419 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:12.947602 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.947609 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.948531 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.948534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.949435 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.949445 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.950092 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.950096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.950211 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.950732 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:12.950748 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:12.953919 (+ 3171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":293,"threads_started":1}
W20260925 15:08:12.954334 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:12.946234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:38065 (local address 127.4.205.254:42671)
0925 15:08:12.946448 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:12.946456 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:12.946784 (+ 328us) server_negotiation.cc:407] Connection header received
0925 15:08:12.946889 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:12.946893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:12.946948 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:12.947022 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:12.947717 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.948410 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.949551 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.949981 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.950066 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.954092 (+ 4026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.193:38065: BlockingRecv error: recv got EOF from 127.4.205.193:38065 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:12 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.195@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:12.999753 14195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:12.991045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.195:58913)
0925 15:08:12.991465 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:12.991481 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:12.991629 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:12.992018 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:12.992025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:12.992571 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:12.992785 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.992799 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.994027 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.994031 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.994892 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:12.994902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.995485 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.995489 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:12.995598 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.996249 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:12.996266 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:12.999420 (+ 3154us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":223,"threads_started":1}
W20260925 15:08:12.999862 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:12.991132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:58913 (local address 127.4.205.254:42671)
0925 15:08:12.991421 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:12.991429 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:12.991579 (+ 150us) server_negotiation.cc:407] Connection header received
0925 15:08:12.991786 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:12.991791 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:12.991868 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:12.991979 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:12.992953 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.993910 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.995016 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:12.995375 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:12.995464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:12.999624 (+ 4160us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.195:58913: BlockingRecv error: recv got EOF from 127.4.205.195:58913 (error 108)
Metrics: {"server-negotiator.queue_time_us":169}
Sep 25 15:08:13 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 15:08:13 dist-test-slave-64fx krb5kdc[13143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.4.205.194@KRBTEST.COM for oryx/127.4.205.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 15:08:13.465075 14196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.456207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42671 (local address 127.4.205.194:60167)
0925 15:08:13.456686 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:13.456701 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:13.456799 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:13.457078 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:13.457081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:13.457411 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:13.457702 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.457713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.458596 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.458599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.459760 (+ 1161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:13.459774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.460657 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.460661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:13.460783 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.461363 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:13.461380 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:13.464681 (+ 3301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.4.205.254:42671: Server oryx/127.4.205.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":262,"threads_started":1}
W20260925 15:08:13.465132 14170 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:13.456253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:60167 (local address 127.4.205.254:42671)
0925 15:08:13.456400 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.456403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.456777 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:08:13.456883 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.456886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.456940 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.457028 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.457812 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.458501 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.459906 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.460521 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.460631 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.464882 (+ 4251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.4.205.194:60167: BlockingRecv error: recv got EOF from 127.4.205.194:60167 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:08:13.762988 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13730
I20260925 15:08:13.774832 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.763919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.4.205.193:39613)
0925 15:08:13.764602 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.764609 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.764634 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:08:13.764696 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.764702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.764927 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.765090 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.766400 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.767573 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.768852 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.769346 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.769505 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.771780 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.771788 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.771801 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.771843 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.774312 (+ 2469us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:08:13.774583 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55486: Request ticket server kudu/127.4.205.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":357,"threads_started":1}
W20260925 15:08:13.774945 14198 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55486: Request ticket server kudu/127.4.205.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:08:13.774938 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13730: Not authorized: Client connection negotiation failed: client connection to 127.4.205.193:39613: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:08:13.775141 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13730
I20260925 15:08:13.791677 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13881
I20260925 15:08:13.804163 14199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.792804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51862 (local address 127.4.205.194:36953)
0925 15:08:13.793401 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.793410 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.793437 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:08:13.793517 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.793524 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.793859 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.794069 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.795524 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.796602 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.797865 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.798439 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.798595 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.800932 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.800942 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.800956 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.801017 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.803463 (+ 2446us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:08:13.803810 (+ 347us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51862: Request ticket server kudu/127.4.205.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
W20260925 15:08:13.804174 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13881: Not authorized: Client connection negotiation failed: client connection to 127.4.205.194:36953: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:08:13.804276 14199 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51862: Request ticket server kudu/127.4.205.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:08:13.804317 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13881
I20260925 15:08:13.820381 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14014
I20260925 15:08:13.832693 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:13.821412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35342 (local address 127.4.205.195:35885)
0925 15:08:13.822101 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:13.822110 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:13.822140 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:08:13.822211 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:13.822220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:13.822529 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:13.822743 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:13.823924 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.825447 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.826652 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:13.827185 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:13.827283 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:13.829497 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:13.829505 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:13.829518 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:13.829553 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:13.832169 (+ 2616us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 15:08:13.832436 (+ 267us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35342: Request ticket server kudu/127.4.205.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":348,"threads_started":1}
W20260925 15:08:13.832722 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14014: Not authorized: Client connection negotiation failed: client connection to 127.4.205.195:35885: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.4.205.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 15:08:13.832801 14200 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35342: Request ticket server kudu/127.4.205.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 15:08:13.832878 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14014
I20260925 15:08:13.849059 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 13661
I20260925 15:08:13.850255 13723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:13.962379 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 13661
2026-09-25T15:08:13Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21062 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T15:08:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:14Z Disabled control of system clock
I20260925 15:08:14.032564 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44671
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44671
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 15:08:14.338393 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:14.338670 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:14.338825 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:14.345714 14205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:14.345808 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:14.345911 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:14.346017 14205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:14.346093 14205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:14.354795 14205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:14.355995 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:14.357674 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:14.368803 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:14.368850 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:14.369215 14205 server_base.cc:1034] running on GCE node
W20260925 15:08:14.368891 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:14.370198 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:14.371392 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:14.372591 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790348894372505 us; error 54 us; skew 500 ppm
I20260925 15:08:14.373013 14205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:14.375515 14205 webserver.cc:460] Webserver started at http://127.4.205.254:43023/ using document root <none> and password file <none>
I20260925 15:08:14.376096 14205 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:14.376173 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:14.376425 14205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:14.379029 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "34f734156fcb4222a874159cbb5bbcea"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "19d3ce422d00f4fc04d353906f320780"
server_key_iv: "f4499060dd1bdaa4c6db44cd2e8e6196"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.379685 14205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "34f734156fcb4222a874159cbb5bbcea"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "19d3ce422d00f4fc04d353906f320780"
server_key_iv: "f4499060dd1bdaa4c6db44cd2e8e6196"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.384312 14205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:08:14.387424 14219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:14.388645 14205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:08:14.388810 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "34f734156fcb4222a874159cbb5bbcea"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "19d3ce422d00f4fc04d353906f320780"
server_key_iv: "f4499060dd1bdaa4c6db44cd2e8e6196"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.389003 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:14.406412 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:14.407182 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:14.407461 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:14.424710 14205 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44671
I20260925 15:08:14.424728 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44671 every 8 connection(s)
I20260925 15:08:14.426090 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:14.429493 14271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:14.430573 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14205
I20260925 15:08:14.430758 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:08:14.431371 4919 external_mini_cluster.cc:1428] Setting key 33f9e468072aded62ef979ba45182daa
I20260925 15:08:14.443785 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea: Bootstrap starting.
I20260925 15:08:14.446095 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.433247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.4.205.254:44671)
0925 15:08:14.434355 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.434369 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.434418 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:08:14.436024 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.436050 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.436654 (+ 604us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.437081 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:14.437613 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.439462 (+ 1849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.440853 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.441968 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.442136 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.443224 (+ 1088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:14.443234 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:14.443253 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:14.443981 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:14.444092 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:14.444119 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:14.444227 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.444396 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.444726 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":343,"threads_started":1}
I20260925 15:08:14.447460 14271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:14.448467 14271 log.cc:824] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:14.451079 14271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea: No bootstrap required, opened a new log
I20260925 15:08:14.458485 14271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } }
I20260925 15:08:14.458796 14271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:14.458865 14271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f734156fcb4222a874159cbb5bbcea, State: Initialized, Role: FOLLOWER
I20260925 15:08:14.459342 14271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } }
I20260925 15:08:14.459530 14271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:14.459621 14271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:14.459738 14271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:14.461947 14271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } }
I20260925 15:08:14.462508 14271 leader_election.cc:302] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f734156fcb4222a874159cbb5bbcea; no voters:
I20260925 15:08:14.463052 14271 leader_election.cc:288] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:08:14.463191 14276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:14.463935 14276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [term 1 LEADER]: Becoming Leader. State: Replica: 34f734156fcb4222a874159cbb5bbcea, State: Running, Role: LEADER
I20260925 15:08:14.464506 14276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } }
I20260925 15:08:14.465044 14271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:14.467128 14278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f734156fcb4222a874159cbb5bbcea. Latest consensus state: current_term: 1 leader_uuid: "34f734156fcb4222a874159cbb5bbcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } } }
I20260925 15:08:14.467415 14278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:14.468235 14283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:14.468104 14277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f734156fcb4222a874159cbb5bbcea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f734156fcb4222a874159cbb5bbcea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44671 } } }
I20260925 15:08:14.468464 14277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:14.472790 14283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:14.481003 14283 catalog_manager.cc:1347] Generated new cluster ID: a8f2bdc2a8b74f8f86df31273f74a270
I20260925 15:08:14.481122 14283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:14.496687 14283 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:08:14.497959 14283 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:14.504365 14283 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f734156fcb4222a874159cbb5bbcea: Generated new TSK 0
I20260925 15:08:14.505141 14283 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:08:14.518532 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--builtin_ntp_servers=127.4.205.212:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 15:08:14.828569 14295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:14.828819 14295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:14.828898 14295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:14.836220 14295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:14.836313 14295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:14.836516 14295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:08:14.845527 14295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:14.846825 14295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:14.848485 14295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:14.861019 14300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:14.861071 14303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:14.861050 14301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:14.861744 14295 server_base.cc:1034] running on GCE node
I20260925 15:08:14.862231 14295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:14.863041 14295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:14.864214 14295 hybrid_clock.cc:648] HybridClock initialized: now 1790348894864161 us; error 55 us; skew 500 ppm
I20260925 15:08:14.864459 14295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:14.866608 14295 webserver.cc:460] Webserver started at http://127.4.205.193:41937/ using document root <none> and password file <none>
I20260925 15:08:14.867255 14295 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:14.867334 14295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:14.867581 14295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:14.870226 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "ebe4990a77c04298bbc3d5015f280071"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "ded617380b7d19dcf159902bf1117b9f"
server_key_iv: "b96349ddc3b0812ee681a9c7f2329b89"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.870919 14295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "ebe4990a77c04298bbc3d5015f280071"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "ded617380b7d19dcf159902bf1117b9f"
server_key_iv: "b96349ddc3b0812ee681a9c7f2329b89"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.875568 14295 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:14.878674 14309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:14.879851 14295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:08:14.879971 14295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "ebe4990a77c04298bbc3d5015f280071"
format_stamp: "Formatted at 2026-09-25 15:08:14 on dist-test-slave-64fx"
server_key: "ded617380b7d19dcf159902bf1117b9f"
server_key_iv: "b96349ddc3b0812ee681a9c7f2329b89"
server_key_version: "encryptionkey@0"
I20260925 15:08:14.880129 14295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:14.901108 14295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:14.901952 14295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:14.902199 14295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:14.903023 14295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:14.904378 14295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:14.904462 14295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:14.904538 14295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:14.904590 14295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:14.935693 14295 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:44089
I20260925 15:08:14.935788 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:44089 every 8 connection(s)
I20260925 15:08:14.936898 14295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:14.938194 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14295
I20260925 15:08:14.938400 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:08:14.939092 4919 external_mini_cluster.cc:1428] Setting key f4fc3d12215733f6db73ba01db3b51b5
I20260925 15:08:14.941952 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--builtin_ntp_servers=127.4.205.212:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:08:14.950605 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.939429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:47375 (local address 127.4.205.254:44671)
0925 15:08:14.939627 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:14.939633 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:14.941509 (+ 1876us) server_negotiation.cc:407] Connection header received
0925 15:08:14.943347 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:14.943352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:14.943432 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:14.943528 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:14.944515 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.945511 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.947961 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.948339 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.948467 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.949813 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:14.949817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:14.949820 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:14.949843 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:14.949875 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:14.949886 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:14.949980 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:14.950222 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:08:14.950310 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:08:14.951556 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:14.940017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44671 (local address 127.4.205.193:47375)
0925 15:08:14.941363 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:14.941398 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:14.943095 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:14.943767 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:14.943775 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:14.943854 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:14.944350 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:14.944375 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.945725 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.945734 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:14.947820 (+ 2086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:14.947832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:14.948556 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:14.948563 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:14.948769 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:14.949420 (+ 651us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:14.949423 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:14.949457 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:14.949549 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:14.949985 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:14.950016 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:14.950024 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:08:14.950212 (+ 188us) client_negotiation.cc:240] Negotiation successful
0925 15:08:14.950561 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":626,"threads_started":1}
I20260925 15:08:14.952955 14422 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44671
I20260925 15:08:14.953234 14422 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:14.953946 14422 heartbeater.cc:511] Master 127.4.205.254:44671 requested a full tablet report, sending...
I20260925 15:08:14.955964 14236 ts_manager.cc:194] Registered new tserver with Master: ebe4990a77c04298bbc3d5015f280071 (127.4.205.193:44089)
I20260925 15:08:14.958250 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.193:47375
W20260925 15:08:15.248853 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:15.249111 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:15.249191 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:15.256554 14425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:15.256645 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:15.256739 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:08:15.265218 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:15.266479 14425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:15.268050 14425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:15.280090 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:15.280148 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:15.280128 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:15.281082 14425 server_base.cc:1034] running on GCE node
I20260925 15:08:15.281739 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:15.282712 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:15.283929 14425 hybrid_clock.cc:648] HybridClock initialized: now 1790348895283883 us; error 59 us; skew 500 ppm
I20260925 15:08:15.284291 14425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:15.286514 14425 webserver.cc:460] Webserver started at http://127.4.205.194:42639/ using document root <none> and password file <none>
I20260925 15:08:15.287189 14425 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:15.287329 14425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:15.287635 14425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:15.290328 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "bdc58c656a2f4cc293406c188c118cb9"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "0822e834d4549958e9ebab4f5c4d0473"
server_key_iv: "3f44e3385daf73d572f183ff5f9265a2"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.291067 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "bdc58c656a2f4cc293406c188c118cb9"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "0822e834d4549958e9ebab4f5c4d0473"
server_key_iv: "3f44e3385daf73d572f183ff5f9265a2"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.295830 14425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:15.299039 14439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.300163 14425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:08:15.300328 14425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "bdc58c656a2f4cc293406c188c118cb9"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "0822e834d4549958e9ebab4f5c4d0473"
server_key_iv: "3f44e3385daf73d572f183ff5f9265a2"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.300544 14425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:15.318056 14425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:15.318805 14425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:15.319090 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:15.319901 14425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:15.321197 14425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:15.321379 14425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.321506 14425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:15.321619 14425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.353240 14425 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:35795
I20260925 15:08:15.353286 14551 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:35795 every 8 connection(s)
I20260925 15:08:15.354496 14425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:15.363965 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14425
I20260925 15:08:15.364140 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:08:15.364516 4919 external_mini_cluster.cc:1428] Setting key 2208c21efe7eb372c3c1816576672e59
I20260925 15:08:15.366240 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.356771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:39005 (local address 127.4.205.254:44671)
0925 15:08:15.356921 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:15.356925 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:15.358236 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 15:08:15.359906 (+ 1670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:15.359911 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:15.359991 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:15.360106 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:15.361121 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.361944 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.363396 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.363795 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.363921 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.365417 (+ 1496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:15.365423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:15.365428 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:15.365452 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:15.365490 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:15.365505 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:15.365616 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:15.365858 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:08:15.365942 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:08:15.367198 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.357239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44671 (local address 127.4.205.194:39005)
0925 15:08:15.358105 (+ 866us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:15.358136 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:15.359649 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:15.360352 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:15.360360 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:15.360440 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:15.360954 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:15.360968 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.362105 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.362110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:15.363232 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:15.363246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.364018 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.364025 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:15.364306 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.365090 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:15.365094 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:15.365138 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:15.365222 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:15.365633 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:15.365640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:15.365646 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:08:15.365837 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:08:15.366190 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":297,"threads_started":1}
I20260925 15:08:15.367235 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--builtin_ntp_servers=127.4.205.212:40473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 15:08:15.368546 14552 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44671
I20260925 15:08:15.368801 14552 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:15.369431 14552 heartbeater.cc:511] Master 127.4.205.254:44671 requested a full tablet report, sending...
I20260925 15:08:15.370745 14236 ts_manager.cc:194] Registered new tserver with Master: bdc58c656a2f4cc293406c188c118cb9 (127.4.205.194:35795)
I20260925 15:08:15.371937 14236 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.194:39005
W20260925 15:08:15.672431 14555 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:15.672711 14555 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:15.672788 14555 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:15.679628 14555 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:15.679713 14555 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:15.679808 14555 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:08:15.688544 14555 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44671
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:15.689997 14555 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:15.691727 14555 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:15.703914 14560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:15.703976 14563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:15.704015 14561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:15.704845 14555 server_base.cc:1034] running on GCE node
I20260925 15:08:15.705394 14555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:15.706295 14555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:15.707479 14555 hybrid_clock.cc:648] HybridClock initialized: now 1790348895707445 us; error 30 us; skew 500 ppm
I20260925 15:08:15.707752 14555 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:15.715680 14555 webserver.cc:460] Webserver started at http://127.4.205.195:34243/ using document root <none> and password file <none>
I20260925 15:08:15.716317 14555 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:15.716426 14555 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:15.716747 14555 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:15.719391 14555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "90f60e0a5873410eb00efb4b3c98b8fa"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "4c716332cd816d1994bb00892b04b13e"
server_key_iv: "1bc0a060a4c050a9ff4250d6d405ae82"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.720108 14555 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "90f60e0a5873410eb00efb4b3c98b8fa"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "4c716332cd816d1994bb00892b04b13e"
server_key_iv: "1bc0a060a4c050a9ff4250d6d405ae82"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.724988 14555 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:08:15.728111 14569 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.729269 14555 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:08:15.729426 14555 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "90f60e0a5873410eb00efb4b3c98b8fa"
format_stamp: "Formatted at 2026-09-25 15:08:15 on dist-test-slave-64fx"
server_key: "4c716332cd816d1994bb00892b04b13e"
server_key_iv: "1bc0a060a4c050a9ff4250d6d405ae82"
server_key_version: "encryptionkey@0"
I20260925 15:08:15.729657 14555 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:15.743698 14555 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:15.744396 14555 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:15.744666 14555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:15.745430 14555 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:15.746690 14555 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:15.746807 14555 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.746945 14555 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:15.747030 14555 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:15.777846 14555 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:42525
I20260925 15:08:15.777921 14681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:42525 every 8 connection(s)
I20260925 15:08:15.779094 14555 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:15.779417 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14555
I20260925 15:08:15.779567 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:08:15.780102 4919 external_mini_cluster.cc:1428] Setting key 665b4918e7ab4733be912aa3012e9b14
I20260925 15:08:15.790668 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.781670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:39965 (local address 127.4.205.254:44671)
0925 15:08:15.781879 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:15.781885 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:15.783529 (+ 1644us) server_negotiation.cc:407] Connection header received
0925 15:08:15.785241 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:15.785244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:15.785303 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:15.785387 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:15.786354 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.787032 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.788424 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.788784 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.788867 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.790084 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:15.790088 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:15.790091 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:15.790105 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:15.790131 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:15.790142 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:15.790207 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:15.790404 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 15:08:15.790464 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:08:15.791534 14684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.782296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44671 (local address 127.4.205.195:39965)
0925 15:08:15.783398 (+ 1102us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:15.783432 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:15.785028 (+ 1596us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:15.785548 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:15.785603 (+ 55us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:15.785704 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:15.786226 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:15.786241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.787151 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.787154 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:15.788309 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:15.788320 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.788939 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.788943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:15.789128 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.789789 (+ 661us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:15.789794 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:15.789848 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:15.789923 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:15.790218 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:15.790225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:15.790229 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:08:15.790359 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:08:15.790594 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":322,"threads_started":1}
I20260925 15:08:15.792732 14682 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44671
I20260925 15:08:15.793017 14682 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:15.793823 14682 heartbeater.cc:511] Master 127.4.205.254:44671 requested a full tablet report, sending...
I20260925 15:08:15.795212 14235 ts_manager.cc:194] Registered new tserver with Master: 90f60e0a5873410eb00efb4b3c98b8fa (127.4.205.195:42525)
I20260925 15:08:15.795984 14235 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.195:39965
I20260925 15:08:15.807154 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:08:15.812777 14274 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:15.811296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51024 (local address 127.4.205.254:44671)
0925 15:08:15.811521 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:15.811530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:15.811937 (+ 407us) server_negotiation.cc:407] Connection header received
0925 15:08:15.812083 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:15.812086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:15.812140 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:15.812213 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:15.812579 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51024: BlockingRecv error: recv got EOF from 127.0.0.1:51024 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:08:15.813112 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14295
I20260925 15:08:15.820022 14691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.814041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.4.205.193:44089)
0925 15:08:15.814530 (+ 489us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:15.814537 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:15.814559 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:08:15.814608 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:15.814613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:15.814805 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:15.814936 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:15.815258 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.816199 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.817384 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.817902 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.817987 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.818865 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:15.818872 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:15.818884 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:15.819414 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:15.819454 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:15.819470 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:15.819551 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:15.819679 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:08:15.819767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":238,"threads_started":1}
I20260925 15:08:15.821111 14417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:15.898948 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14295
I20260925 15:08:15.914357 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14425
I20260925 15:08:15.921713 14693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:15.915100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33296 (local address 127.4.205.194:35795)
0925 15:08:15.915569 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:15.915576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:15.915601 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:08:15.915656 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:15.915662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:15.915870 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:15.916041 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:15.916417 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.917335 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.918576 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:15.919142 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:15.919282 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:15.920270 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:15.920278 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:15.920289 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:15.920811 (+ 522us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:15.920869 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:15.920886 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:15.921024 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:15.921147 (+ 123us) server_negotiation.cc:299] Negotiation successful
0925 15:08:15.921295 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20260925 15:08:15.922914 14547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:15.998466 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14425
I20260925 15:08:16.014704 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14555
I20260925 15:08:16.022746 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:16.015626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.4.205.195:42525)
0925 15:08:16.016201 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:16.016208 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:16.016236 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:08:16.016314 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:16.016320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:16.016545 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:16.016721 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:16.017165 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:16.018284 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:16.019551 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:16.020104 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:16.020239 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:16.021254 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:16.021262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:16.021274 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:16.021931 (+ 657us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:16.021990 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:16.022009 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:16.022131 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:16.022285 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:08:16.022424 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":295,"threads_started":1}
I20260925 15:08:16.023885 14677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:16.103030 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14555
I20260925 15:08:16.118422 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14205
I20260925 15:08:16.119480 14266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:16.214713 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14205
2026-09-25T15:08:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2248 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T15:08:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:16Z Disabled control of system clock
I20260925 15:08:16.279309 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44985
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44985
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:08:16.575332 14701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:16.575585 14701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:16.575675 14701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:16.582335 14701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:16.582427 14701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:16.582471 14701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:16.582525 14701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:16.582567 14701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:16.591037 14701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44985
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:16.592144 14701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:16.593837 14701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:16.604934 14706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:16.604990 14707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:16.605028 14709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:16.605775 14701 server_base.cc:1034] running on GCE node
I20260925 15:08:16.606534 14701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:16.607842 14701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:16.609062 14701 hybrid_clock.cc:648] HybridClock initialized: now 1790348896609013 us; error 58 us; skew 500 ppm
I20260925 15:08:16.609382 14701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:16.611495 14701 webserver.cc:460] Webserver started at http://127.4.205.254:42099/ using document root <none> and password file <none>
I20260925 15:08:16.612147 14701 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:16.612273 14701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:16.612612 14701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:16.615284 14701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "e73de1321afb4fd492971c9a2dc4ee80"
format_stamp: "Formatted at 2026-09-25 15:08:16 on dist-test-slave-64fx"
server_key: "cd98f52509a978212546c73493f72a9f"
server_key_iv: "09d00949ef8a642b002d9179ded97d98"
server_key_version: "encryptionkey@0"
I20260925 15:08:16.616030 14701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "e73de1321afb4fd492971c9a2dc4ee80"
format_stamp: "Formatted at 2026-09-25 15:08:16 on dist-test-slave-64fx"
server_key: "cd98f52509a978212546c73493f72a9f"
server_key_iv: "09d00949ef8a642b002d9179ded97d98"
server_key_version: "encryptionkey@0"
I20260925 15:08:16.620503 14701 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:08:16.623575 14715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:16.624723 14701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:08:16.624892 14701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "e73de1321afb4fd492971c9a2dc4ee80"
format_stamp: "Formatted at 2026-09-25 15:08:16 on dist-test-slave-64fx"
server_key: "cd98f52509a978212546c73493f72a9f"
server_key_iv: "09d00949ef8a642b002d9179ded97d98"
server_key_version: "encryptionkey@0"
I20260925 15:08:16.625118 14701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:16.630390 14701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:16.631049 14701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:16.631320 14701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:16.648138 14701 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44985
I20260925 15:08:16.648170 14766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44985 every 8 connection(s)
I20260925 15:08:16.649403 14701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:16.652779 14767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:16.654502 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14701
I20260925 15:08:16.654719 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:08:16.655282 4919 external_mini_cluster.cc:1428] Setting key e7b2df0f2383520b0f6ced1eb9dd00b5
I20260925 15:08:16.663949 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:16.657126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.4.205.254:44985)
0925 15:08:16.658199 (+ 1073us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:16.658213 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:16.658259 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:08:16.660331 (+ 2072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:16.660355 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:16.660822 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:16.661204 (+ 382us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:16.662200 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:16.662208 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:16.662222 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:16.662238 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:16.662310 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:16.662329 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:16.662557 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:16.662697 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:08:16.662929 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":393,"threads_started":1}
I20260925 15:08:16.665033 14767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80: Bootstrap starting.
I20260925 15:08:16.667647 14767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:16.668520 14767 log.cc:824] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:16.670830 14767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80: No bootstrap required, opened a new log
I20260925 15:08:16.677603 14767 raft_consensus.cc:348] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } }
I20260925 15:08:16.678011 14767 raft_consensus.cc:374] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:16.678112 14767 raft_consensus.cc:729] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e73de1321afb4fd492971c9a2dc4ee80, State: Initialized, Role: FOLLOWER
I20260925 15:08:16.678644 14767 consensus_queue.cc:260] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } }
I20260925 15:08:16.678864 14767 raft_consensus.cc:388] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:16.679003 14767 raft_consensus.cc:482] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:16.679167 14767 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:16.681156 14767 raft_consensus.cc:504] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } }
I20260925 15:08:16.681746 14767 leader_election.cc:302] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e73de1321afb4fd492971c9a2dc4ee80; no voters:
I20260925 15:08:16.682312 14767 leader_election.cc:288] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:08:16.682399 14772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:16.683115 14772 raft_consensus.cc:686] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [term 1 LEADER]: Becoming Leader. State: Replica: e73de1321afb4fd492971c9a2dc4ee80, State: Running, Role: LEADER
I20260925 15:08:16.683570 14772 consensus_queue.cc:237] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } }
I20260925 15:08:16.684172 14767 sys_catalog.cc:564] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:16.686385 14774 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e73de1321afb4fd492971c9a2dc4ee80. Latest consensus state: current_term: 1 leader_uuid: "e73de1321afb4fd492971c9a2dc4ee80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } } }
I20260925 15:08:16.686780 14774 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:16.687523 14773 sys_catalog.cc:455] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e73de1321afb4fd492971c9a2dc4ee80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e73de1321afb4fd492971c9a2dc4ee80" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44985 } } }
I20260925 15:08:16.687899 14773 sys_catalog.cc:458] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80 [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:16.690451 14782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:16.694873 14782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:16.703999 14782 catalog_manager.cc:1347] Generated new cluster ID: 8d3357cf7ffc48e080f61328ff2267bf
I20260925 15:08:16.704107 14782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:16.733656 14782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:08:16.734011 14782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:16.745527 14782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e73de1321afb4fd492971c9a2dc4ee80: Generated new TSK 0
I20260925 15:08:16.746227 14782 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:08:16.755791 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--builtin_ntp_servers=127.4.205.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 15:08:17.060515 14791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:17.060737 14791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:17.060817 14791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:17.067543 14791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:17.067620 14791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:17.067745 14791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:08:17.075933 14791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:17.077025 14791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:17.078655 14791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:17.090615 14799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.090724 14796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.090669 14797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:17.091645 14791 server_base.cc:1034] running on GCE node
I20260925 15:08:17.092188 14791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:17.093032 14791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:17.094244 14791 hybrid_clock.cc:648] HybridClock initialized: now 1790348897094214 us; error 46 us; skew 500 ppm
I20260925 15:08:17.094535 14791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:17.096820 14791 webserver.cc:460] Webserver started at http://127.4.205.193:36107/ using document root <none> and password file <none>
I20260925 15:08:17.097446 14791 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:17.097589 14791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:17.097919 14791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:17.102217 14791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "4ca27ce0fda143a09f2282b33dd8b179"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "755548e5bbf39fcbf61eca888c1d1cb9"
server_key_iv: "64478ee820e18f9dd84ed144883aeed2"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.102964 14791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "4ca27ce0fda143a09f2282b33dd8b179"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "755548e5bbf39fcbf61eca888c1d1cb9"
server_key_iv: "64478ee820e18f9dd84ed144883aeed2"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.107658 14791 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:17.110697 14805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.111830 14791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:08:17.111994 14791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "4ca27ce0fda143a09f2282b33dd8b179"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "755548e5bbf39fcbf61eca888c1d1cb9"
server_key_iv: "64478ee820e18f9dd84ed144883aeed2"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.112195 14791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:17.117282 14791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:17.118068 14791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:17.118316 14791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:17.119140 14791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:17.120385 14791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:17.120502 14791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.120630 14791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:17.120711 14791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.151708 14791 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:44977
I20260925 15:08:17.151788 14917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:44977 every 8 connection(s)
I20260925 15:08:17.152993 14791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:17.157316 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14791
I20260925 15:08:17.157541 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:08:17.158133 4919 external_mini_cluster.cc:1428] Setting key 5f7f62cf91d9b5e1dc34e0a2a6373693
I20260925 15:08:17.160957 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--builtin_ntp_servers=127.4.205.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:08:17.161794 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.155551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50457 (local address 127.4.205.254:44985)
0925 15:08:17.155728 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:17.155733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:17.157112 (+ 1379us) server_negotiation.cc:407] Connection header received
0925 15:08:17.159249 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:17.159252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:17.159295 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:17.159372 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:17.160880 (+ 1508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:17.160889 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:17.160893 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:17.160911 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:17.160954 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:17.160970 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:17.161103 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:17.161365 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:08:17.161440 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:08:17.162582 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.156011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44985 (local address 127.4.205.193:50457)
0925 15:08:17.156949 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:17.156990 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:17.158984 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:17.159602 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:17.159612 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:17.159725 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:17.160578 (+ 853us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:17.160580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:17.160614 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:17.160708 (+ 94us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:17.161109 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:17.161120 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:17.161130 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 15:08:17.161426 (+ 296us) client_negotiation.cc:240] Negotiation successful
0925 15:08:17.161666 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":264,"threads_started":1}
I20260925 15:08:17.163971 14918 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44985
I20260925 15:08:17.164247 14918 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:17.164749 14918 heartbeater.cc:511] Master 127.4.205.254:44985 requested a full tablet report, sending...
I20260925 15:08:17.166587 14732 ts_manager.cc:194] Registered new tserver with Master: 4ca27ce0fda143a09f2282b33dd8b179 (127.4.205.193:44977)
W20260925 15:08:17.476712 14921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:17.476948 14921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:17.477038 14921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:17.483927 14921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:17.484014 14921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:17.484210 14921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:08:17.493003 14921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:17.494299 14921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:17.496024 14921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:17.508091 14927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.508132 14926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.508172 14929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:17.508844 14921 server_base.cc:1034] running on GCE node
I20260925 15:08:17.509349 14921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:17.510270 14921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:17.511461 14921 hybrid_clock.cc:648] HybridClock initialized: now 1790348897511424 us; error 31 us; skew 500 ppm
I20260925 15:08:17.511727 14921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:17.513907 14921 webserver.cc:460] Webserver started at http://127.4.205.194:35439/ using document root <none> and password file <none>
I20260925 15:08:17.514497 14921 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:17.514576 14921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:17.514819 14921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:17.517445 14921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "8ad1d455e57846e186448c365b8b76d3"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "893ed9bdc6b6d1ec10753e9dd7ed2c2d"
server_key_iv: "47babdca69077a1cae7c27221b752d80"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.518211 14921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "8ad1d455e57846e186448c365b8b76d3"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "893ed9bdc6b6d1ec10753e9dd7ed2c2d"
server_key_iv: "47babdca69077a1cae7c27221b752d80"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.522732 14921 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:08:17.525775 14935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.526906 14921 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:08:17.527030 14921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "8ad1d455e57846e186448c365b8b76d3"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "893ed9bdc6b6d1ec10753e9dd7ed2c2d"
server_key_iv: "47babdca69077a1cae7c27221b752d80"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.527185 14921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:17.532234 14921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:17.532874 14921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:17.533087 14921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:17.533913 14921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:17.535116 14921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:17.535195 14921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.535269 14921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:17.535321 14921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.566201 14921 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:34511
I20260925 15:08:17.566277 15047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:34511 every 8 connection(s)
I20260925 15:08:17.567360 14921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:17.571148 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 14921
I20260925 15:08:17.571329 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:08:17.571858 4919 external_mini_cluster.cc:1428] Setting key a314f397ec9cfbc63a5f14b7fdc70607
I20260925 15:08:17.574839 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--builtin_ntp_servers=127.4.205.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 15:08:17.575465 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.569590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:42161 (local address 127.4.205.254:44985)
0925 15:08:17.569758 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:17.569762 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:17.570961 (+ 1199us) server_negotiation.cc:407] Connection header received
0925 15:08:17.573119 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:17.573124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:17.573173 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:17.573295 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:17.574669 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:17.574673 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:17.574676 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:17.574690 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:17.574719 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:17.574730 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:17.574814 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:17.575084 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:08:17.575142 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:08:17.577023 15050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.569999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44985 (local address 127.4.205.194:42161)
0925 15:08:17.570771 (+ 772us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:17.570836 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:17.572877 (+ 2041us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:17.573518 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:17.573529 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:17.573659 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:17.574335 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:17.574339 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:17.574385 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:17.574485 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:17.574830 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:17.574840 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:17.574848 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:08:17.575725 (+ 877us) client_negotiation.cc:240] Negotiation successful
0925 15:08:17.575960 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":238,"threads_started":1}
I20260925 15:08:17.578235 15048 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44985
I20260925 15:08:17.578502 15048 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:17.579003 15048 heartbeater.cc:511] Master 127.4.205.254:44985 requested a full tablet report, sending...
I20260925 15:08:17.580282 14732 ts_manager.cc:194] Registered new tserver with Master: 8ad1d455e57846e186448c365b8b76d3 (127.4.205.194:34511)
W20260925 15:08:17.881868 15051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:17.882134 15051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:17.882218 15051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:17.889084 15051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:17.889165 15051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:17.889258 15051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:08:17.897670 15051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44985
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:17.898734 15051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:17.900349 15051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:17.912761 15056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.912904 15057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:17.912992 15059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:17.913882 15051 server_base.cc:1034] running on GCE node
I20260925 15:08:17.914408 15051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:17.915292 15051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:17.916489 15051 hybrid_clock.cc:648] HybridClock initialized: now 1790348897916435 us; error 41 us; skew 500 ppm
I20260925 15:08:17.916791 15051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:08:17.918936 15051 webserver.cc:460] Webserver started at http://127.4.205.195:37949/ using document root <none> and password file <none>
I20260925 15:08:17.919577 15051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:17.919695 15051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:17.920006 15051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:17.922614 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "910b8f2c4f9643b58f3a9600cb4d8470"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "ff9dfda861b9c50f3cf24da79724b95d"
server_key_iv: "4f5eee665389d4eccfc33df352025f1c"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.923374 15051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "910b8f2c4f9643b58f3a9600cb4d8470"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "ff9dfda861b9c50f3cf24da79724b95d"
server_key_iv: "4f5eee665389d4eccfc33df352025f1c"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.928045 15051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:17.931424 15065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.932686 15051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:08:17.932811 15051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "910b8f2c4f9643b58f3a9600cb4d8470"
format_stamp: "Formatted at 2026-09-25 15:08:17 on dist-test-slave-64fx"
server_key: "ff9dfda861b9c50f3cf24da79724b95d"
server_key_iv: "4f5eee665389d4eccfc33df352025f1c"
server_key_version: "encryptionkey@0"
I20260925 15:08:17.932969 15051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:17.938120 15051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:17.938762 15051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:17.938952 15051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:17.939713 15051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:17.940888 15051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:17.940966 15051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.941069 15051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:17.941123 15051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:17.971918 15051 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34693
I20260925 15:08:17.971967 15177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34693 every 8 connection(s)
I20260925 15:08:17.973199 15051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:17.975463 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 15051
I20260925 15:08:17.975656 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:08:17.976053 4919 external_mini_cluster.cc:1428] Setting key d5b7d7824b93ef2516d8678dbd0e9377
I20260925 15:08:17.984935 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.975619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:50447 (local address 127.4.205.254:44985)
0925 15:08:17.975830 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:17.975835 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:17.977222 (+ 1387us) server_negotiation.cc:407] Connection header received
0925 15:08:17.978860 (+ 1638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:17.978865 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:17.978927 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:17.979039 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:17.984219 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:17.984226 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:17.984238 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:17.984253 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:17.984281 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:17.984293 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:17.984430 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:17.984603 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 15:08:17.984665 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:08:17.985859 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.976166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44985 (local address 127.4.205.195:50447)
0925 15:08:17.977088 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:17.977135 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:17.978630 (+ 1495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:17.979237 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:17.979244 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:17.979324 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:17.983949 (+ 4625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:08:17.983952 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:17.983987 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:08:17.984067 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:17.984383 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:17.984393 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:17.984400 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:08:17.984670 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 15:08:17.984960 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":291,"threads_started":1}
I20260925 15:08:17.987041 15178 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44985
I20260925 15:08:17.987272 15178 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:17.987730 15178 heartbeater.cc:511] Master 127.4.205.254:44985 requested a full tablet report, sending...
I20260925 15:08:17.989017 14732 ts_manager.cc:194] Registered new tserver with Master: 910b8f2c4f9643b58f3a9600cb4d8470 (127.4.205.195:34693)
I20260925 15:08:17.990872 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 15:08:17.996663 14770 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:08:17.995140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46742 (local address 127.4.205.254:44985)
0925 15:08:17.995292 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:17.995295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:17.995768 (+ 473us) server_negotiation.cc:407] Connection header received
0925 15:08:17.995974 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:17.995980 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:17.996036 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:17.996142 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:17.996438 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46742: BlockingRecv error: recv got EOF from 127.0.0.1:46742 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:08:17.997136 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14791
I20260925 15:08:18.000175 15187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:17.998028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53314 (local address 127.4.205.193:44977)
0925 15:08:17.998516 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:17.998522 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:17.998545 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:08:17.998598 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:17.998603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:17.998762 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:17.998875 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:17.999652 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:17.999658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:17.999666 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:17.999682 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:17.999715 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:17.999727 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:17.999822 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:17.999900 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:08:17.999965 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":235,"threads_started":1}
I20260925 15:08:18.001209 14913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:18.078068 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14791
I20260925 15:08:18.093523 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14921
I20260925 15:08:18.097155 15189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:18.094498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40908 (local address 127.4.205.194:34511)
0925 15:08:18.095000 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:18.095007 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:18.095031 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:08:18.095087 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:18.095093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:18.095326 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:18.095494 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:18.096487 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:18.096493 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:18.096504 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:18.096520 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:18.096557 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:18.096569 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:18.096717 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:18.096815 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:08:18.096909 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":245,"threads_started":1}
I20260925 15:08:18.098376 15043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:18.176718 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14921
I20260925 15:08:18.195465 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 15051
I20260925 15:08:18.200392 15191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:18.196642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34336 (local address 127.4.205.195:34693)
0925 15:08:18.197459 (+ 817us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:18.197466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:18.197493 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:08:18.197551 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:18.197606 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:18.197949 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:18.198130 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:18.199483 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:18.199492 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:18.199509 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:08:18.199525 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:18.199587 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:08:18.199609 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:18.199851 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:18.199930 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:08:18.200040 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":452,"threads_started":1}
I20260925 15:08:18.201840 15173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:18.292089 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 15051
I20260925 15:08:18.307624 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 14701
I20260925 15:08:18.308849 14762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:18.410111 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 14701
2026-09-25T15:08:18Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2195 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:08:18 dist-test-slave-64fx krb5kdc[15196](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:08:18 dist-test-slave-64fx krb5kdc[15196](info): set up 2 sockets
Sep 25 15:08:18 dist-test-slave-64fx krb5kdc[15196](info): commencing operation
krb5kdc: starting...
W20260925 15:08:19.474442 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:08:19 dist-test-slave-64fx krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:08:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:08:19.680609 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40081
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:40081
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:19.979058 15209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:19.979323 15209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:19.979481 15209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:19.986014 15209 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:08:19.986124 15209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:19.986220 15209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:19.986312 15209 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:08:19.986375 15209 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:08:19.995190 15209 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:40081
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:19.996317 15209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:19.997527 15209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:20.008562 15214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:20.008533 15215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:20.009006 15217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:20.009301 15209 server_base.cc:1034] running on GCE node
I20260925 15:08:20.010031 15209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:20.011232 15209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:20.012419 15209 hybrid_clock.cc:648] HybridClock initialized: now 1790348900012384 us; error 43 us; skew 500 ppm
I20260925 15:08:20.012697 15209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:20 dist-test-slave-64fx krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348900, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:20.015740 15209 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:08:20.017145 15209 webserver.cc:460] Webserver started at http://127.4.205.254:40505/ using document root <none> and password file <none>
I20260925 15:08:20.017848 15209 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:20.017992 15209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:20.018277 15209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:20.020998 15209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "e0ce684a5cab34828878841287fe0b15"
server_key_iv: "e38705693d3815e99422eae2df242269"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.021800 15209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "e0ce684a5cab34828878841287fe0b15"
server_key_iv: "e38705693d3815e99422eae2df242269"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.026422 15209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 15:08:20.029404 15224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:20.030582 15209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:08:20.030747 15209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "e0ce684a5cab34828878841287fe0b15"
server_key_iv: "e38705693d3815e99422eae2df242269"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.030922 15209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:20.056442 15209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:20.060568 15209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:20.060837 15209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:20.078353 15209 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:40081
I20260925 15:08:20.078387 15275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:40081 every 8 connection(s)
I20260925 15:08:20.079561 15209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:08:20.082989 15276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:20.088232 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 15209
I20260925 15:08:20.088485 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:08:20.089233 4919 external_mini_cluster.cc:1428] Setting key cae4426076811ea8a252ae38add4213f
I20260925 15:08:20.093268 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: Bootstrap starting.
I20260925 15:08:20.096047 15276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:20.097069 15276 log.cc:824] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:20.099632 15276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: No bootstrap required, opened a new log
Sep 25 15:08:20 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:08:20.107297 15276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } }
I20260925 15:08:20.107699 15276 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:20.107805 15276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c36a6ad9c2a43d99e55ecb4bfc546cd, State: Initialized, Role: FOLLOWER
I20260925 15:08:20.108455 15276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } }
I20260925 15:08:20.108721 15276 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:08:20.108866 15276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:08:20.109067 15276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:20.110811 15279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:20.091280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.4.205.254:40081)
0925 15:08:20.092005 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:20.092016 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:20.092053 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:08:20.093705 (+ 1652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:20.093727 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:20.094060 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:20.094411 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:20.095775 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.097226 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.098565 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.099229 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.099411 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:20.103545 (+ 4134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:20.103552 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:20.103563 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:20.103605 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:20.106456 (+ 2851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:20.107136 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:20.107140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:20.107145 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:20.107292 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:20.107650 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:20.107656 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:20.107661 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:20.108828 (+ 1167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:20.109184 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:20.109476 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:08:20.109917 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":347,"threads_started":1}
I20260925 15:08:20.111447 15276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } }
I20260925 15:08:20.111920 15276 leader_election.cc:302] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c36a6ad9c2a43d99e55ecb4bfc546cd; no voters:
I20260925 15:08:20.112485 15276 leader_election.cc:288] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:08:20.112607 15281 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:20.113710 15281 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [term 1 LEADER]: Becoming Leader. State: Replica: 9c36a6ad9c2a43d99e55ecb4bfc546cd, State: Running, Role: LEADER
I20260925 15:08:20.114298 15281 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } }
I20260925 15:08:20.114540 15276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:08:20.117347 15283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c36a6ad9c2a43d99e55ecb4bfc546cd. Latest consensus state: current_term: 1 leader_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } } }
I20260925 15:08:20.117697 15283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:20.117437 15282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c36a6ad9c2a43d99e55ecb4bfc546cd" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40081 } } }
I20260925 15:08:20.117833 15282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd [sys.catalog]: This master's current role is: LEADER
I20260925 15:08:20.128773 15290 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:08:20.133076 15290 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:08:20.140146 15290 catalog_manager.cc:1347] Generated new cluster ID: 328557ff4d674598ae654b9988738008
I20260925 15:08:20.140273 15290 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:08:20.153661 15290 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:08:20.155228 15290 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:08:20.163139 15290 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: Generated new TSK 0
I20260925 15:08:20.163923 15290 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:08:20.223326 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:20.532207 15302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:20.532449 15302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:20.532594 15302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:20.539371 15302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:20.539453 15302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:20.539613 15302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:08:20.548372 15302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:20.549495 15302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:20.550678 15302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:20.562748 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:20.562830 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:20.562775 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:20.563839 15302 server_base.cc:1034] running on GCE node
I20260925 15:08:20.564386 15302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:20.565204 15302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:20.566408 15302 hybrid_clock.cc:648] HybridClock initialized: now 1790348900566364 us; error 57 us; skew 500 ppm
I20260925 15:08:20.566691 15302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:20 dist-test-slave-64fx krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348900, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:20.569626 15302 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:08:20.571022 15302 webserver.cc:460] Webserver started at http://127.4.205.193:42547/ using document root <none> and password file <none>
I20260925 15:08:20.571609 15302 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:20.571751 15302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:20.572033 15302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:20.574659 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "ed1492fcbbb641a38e874ad7ece391aa"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "385ccbeffb0e89853ffdd7759b2fedbe"
server_key_iv: "22f1bcf1477b39e57b6fffa37336d94a"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.575363 15302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "ed1492fcbbb641a38e874ad7ece391aa"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "385ccbeffb0e89853ffdd7759b2fedbe"
server_key_iv: "22f1bcf1477b39e57b6fffa37336d94a"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.579763 15302 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:08:20.582837 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:20.583915 15302 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:08:20.584085 15302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "ed1492fcbbb641a38e874ad7ece391aa"
format_stamp: "Formatted at 2026-09-25 15:08:20 on dist-test-slave-64fx"
server_key: "385ccbeffb0e89853ffdd7759b2fedbe"
server_key_iv: "22f1bcf1477b39e57b6fffa37336d94a"
server_key_version: "encryptionkey@0"
I20260925 15:08:20.584265 15302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:20.597293 15302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:20.601138 15302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:20.601428 15302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:20.602313 15302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:20.603395 15302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:20.603475 15302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:20.603549 15302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:20.603595 15302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:20.635375 15302 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:39909
I20260925 15:08:20.635381 15429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:39909 every 8 connection(s)
I20260925 15:08:20.636533 15302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:08:20.642225 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 15302
I20260925 15:08:20.642444 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:08:20.643095 4919 external_mini_cluster.cc:1428] Setting key 1276e1c5d124a3af15d7fd5fb105c794
Sep 25 15:08:20 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348900, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:08:20.657126 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:20.638800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:37873 (local address 127.4.205.254:40081)
0925 15:08:20.639199 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:20.639203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:20.640110 (+ 907us) server_negotiation.cc:407] Connection header received
0925 15:08:20.641767 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:20.641772 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:20.641877 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:20.642018 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:20.643470 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.644427 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.646143 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.646876 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.647018 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:20.650977 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:20.650985 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:20.650992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:20.651032 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:20.654160 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:20.654793 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:20.654801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:20.654807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:20.654941 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:20.655312 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:20.655319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:20.655324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:20.655684 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:20.655842 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:20.656423 (+ 581us) server_negotiation.cc:299] Negotiation successful
0925 15:08:20.656667 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":216,"threads_started":1}
I20260925 15:08:20.657647 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:20.639355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40081 (local address 127.4.205.193:37873)
0925 15:08:20.639968 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:20.639999 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:20.641480 (+ 1481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:20.642165 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:20.642172 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:20.642727 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:20.643312 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:20.643329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.644571 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.644576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:20.646007 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:20.646018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:20.647066 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:20.647070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:20.647242 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:20.647866 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:20.647885 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:20.650770 (+ 2885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:20.654286 (+ 3516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:20.654293 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:20.654303 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:20.654675 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:20.655036 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:20.655039 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:20.655042 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:20.655198 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:20.655808 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:20.655813 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:20.656080 (+ 267us) client_negotiation.cc:769] Sending connection context
0925 15:08:20.656273 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 15:08:20.656590 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":242,"threads_started":1}
I20260925 15:08:20.659127 15430 heartbeater.cc:347] Connected to a master server at 127.4.205.254:40081
I20260925 15:08:20.659412 15430 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:20.660075 15430 heartbeater.cc:511] Master 127.4.205.254:40081 requested a full tablet report, sending...
I20260925 15:08:20.662312 15241 ts_manager.cc:194] Registered new tserver with Master: ed1492fcbbb641a38e874ad7ece391aa (127.4.205.193:39909)
I20260925 15:08:20.664603 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:37873
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:08:20.691906 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:20.994280 15436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:20.994539 15436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:20.994712 15436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:21.001451 15436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:21.001539 15436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:21.001797 15436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:08:21.011247 15436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:21.012336 15436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:21.013547 15436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:21.026950 15442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:21.026891 15441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:21.027004 15444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:21.027590 15436 server_base.cc:1034] running on GCE node
I20260925 15:08:21.028195 15436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:21.029001 15436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:21.030179 15436 hybrid_clock.cc:648] HybridClock initialized: now 1790348901030124 us; error 47 us; skew 500 ppm
I20260925 15:08:21.030467 15436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:21 dist-test-slave-64fx krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348901, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:21.040185 15436 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:08:21.041538 15436 webserver.cc:460] Webserver started at http://127.4.205.194:45905/ using document root <none> and password file <none>
I20260925 15:08:21.042217 15436 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:21.042356 15436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:21.042634 15436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:21.045270 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "270ca57733ed4c01a7be11da093f69d6"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "ccb33e813f4b81000d77a689b1982e4b"
server_key_iv: "88832837137415953fc2e18f960ec264"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.046016 15436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "270ca57733ed4c01a7be11da093f69d6"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "ccb33e813f4b81000d77a689b1982e4b"
server_key_iv: "88832837137415953fc2e18f960ec264"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.050629 15436 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:08:21.053680 15451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.054778 15436 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:08:21.054939 15436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "270ca57733ed4c01a7be11da093f69d6"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "ccb33e813f4b81000d77a689b1982e4b"
server_key_iv: "88832837137415953fc2e18f960ec264"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.055112 15436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:21.080065 15436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:21.084656 15436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:21.084945 15436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:21.085896 15436 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:21.087442 15436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:21.087544 15436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.087669 15436 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:21.087723 15436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.119583 15436 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:35967
I20260925 15:08:21.119623 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:35967 every 8 connection(s)
I20260925 15:08:21.120764 15436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:08:21.129731 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 15436
I20260925 15:08:21.129917 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:08:21.130365 4919 external_mini_cluster.cc:1428] Setting key e69914ab1561ab2a275d8ca39bb20461
Sep 25 15:08:21 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348901, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:08:21.141050 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.123215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:41757 (local address 127.4.205.254:40081)
0925 15:08:21.123447 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.123452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.124563 (+ 1111us) server_negotiation.cc:407] Connection header received
0925 15:08:21.126337 (+ 1774us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.126342 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.126407 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.126492 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:21.127836 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.128562 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.130302 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.130888 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.130992 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.134538 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:21.134546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:21.134554 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:21.134595 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:21.138065 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.138771 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.138779 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.138786 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.138949 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.139330 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.139337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.139343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.139698 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:21.139831 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.140671 (+ 840us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.140827 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 15:08:21.142601 15566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.123763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40081 (local address 127.4.205.194:41757)
0925 15:08:21.124407 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:21.124448 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:21.126103 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:21.126683 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:21.126690 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:21.127167 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:21.127700 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.127715 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.128677 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.128681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.130170 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.130181 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.131116 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.131127 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.131323 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.132015 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:21.132047 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:21.134308 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:21.138195 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:21.138201 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:21.138212 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:21.138632 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:21.139049 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:21.139052 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:21.139054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:21.139211 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:21.139843 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:21.139854 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:21.140286 (+ 432us) client_negotiation.cc:769] Sending connection context
0925 15:08:21.140600 (+ 314us) client_negotiation.cc:240] Negotiation successful
0925 15:08:21.141149 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":250,"threads_started":1}
I20260925 15:08:21.143916 15564 heartbeater.cc:347] Connected to a master server at 127.4.205.254:40081
I20260925 15:08:21.144207 15564 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:21.144845 15564 heartbeater.cc:511] Master 127.4.205.254:40081 requested a full tablet report, sending...
I20260925 15:08:21.146150 15241 ts_manager.cc:194] Registered new tserver with Master: 270ca57733ed4c01a7be11da093f69d6 (127.4.205.194:35967)
I20260925 15:08:21.146903 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:41757
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:08:21.180001 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:35603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:08:21.482800 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:08:21.483076 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:08:21.483191 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:08:21.490198 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:08:21.490329 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:08:21.490530 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:08:21.499138 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:35603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:40081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:08:21.500240 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:08:21.501442 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:08:21.513980 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:21.513988 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:08:21.514040 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:08:21.514518 15569 server_base.cc:1034] running on GCE node
I20260925 15:08:21.515142 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:08:21.516034 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:08:21.517242 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790348901517205 us; error 54 us; skew 500 ppm
I20260925 15:08:21.517518 15569 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:08:21 dist-test-slave-64fx krb5kdc[15196](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348901, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:08:21.520256 15569 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:08:21.521509 15569 webserver.cc:460] Webserver started at http://127.4.205.195:33857/ using document root <none> and password file <none>
I20260925 15:08:21.522195 15569 fs_manager.cc:359] Metadata directory not provided
I20260925 15:08:21.522332 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:08:21.522609 15569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:08:21.525213 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "1a0fab4c2eab753939b723d2ebc2d643"
server_key_iv: "78913ca3c3f440fec5e76efe7c80e3b1"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.525928 15569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "1a0fab4c2eab753939b723d2ebc2d643"
server_key_iv: "78913ca3c3f440fec5e76efe7c80e3b1"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.530565 15569 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 15:08:21.533524 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.534777 15569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:08:21.534937 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41"
format_stamp: "Formatted at 2026-09-25 15:08:21 on dist-test-slave-64fx"
server_key: "1a0fab4c2eab753939b723d2ebc2d643"
server_key_iv: "78913ca3c3f440fec5e76efe7c80e3b1"
server_key_version: "encryptionkey@0"
I20260925 15:08:21.535113 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:08:21.564949 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:08:21.568593 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:08:21.568854 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:08:21.569749 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:08:21.570899 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:08:21.570992 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.571132 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:08:21.571208 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:08:21.602216 15569 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:41669
I20260925 15:08:21.602264 15696 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:41669 every 8 connection(s)
I20260925 15:08:21.603452 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:08:21.608322 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 15569
I20260925 15:08:21.608526 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:08:21.609045 4919 external_mini_cluster.cc:1428] Setting key 3025816604815f13139d09f8c1e8fc69
Sep 25 15:08:21 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348901, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:08:21.621673 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.605731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:34053 (local address 127.4.205.254:40081)
0925 15:08:21.605936 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.605943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.607017 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 15:08:21.608756 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.608762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.608846 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.608961 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:21.610599 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.611530 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.613167 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.613633 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.613737 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.616876 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:21.616880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:21.616883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:21.616912 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:21.619300 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.619917 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.619921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.619923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.620024 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.620346 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.620348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.620350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.620586 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:21.620684 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.621191 (+ 507us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.621362 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:08:21.622630 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.606243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40081 (local address 127.4.205.195:34053)
0925 15:08:21.606861 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:21.606904 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:21.608414 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:21.609194 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:21.609202 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:21.609846 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:08:21.610439 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.610457 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.611743 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.611749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.613045 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.613057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.613819 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.613824 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.614010 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.614643 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:08:21.614663 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:08:21.616710 (+ 2047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:08:21.619414 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:21.619420 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:21.619429 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:21.619814 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:21.620114 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:08:21.620117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:08:21.620119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:08:21.620265 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:08:21.620678 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:08:21.620683 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:08:21.620923 (+ 240us) client_negotiation.cc:769] Sending connection context
0925 15:08:21.621118 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 15:08:21.621460 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":251,"threads_started":1}
I20260925 15:08:21.623942 15697 heartbeater.cc:347] Connected to a master server at 127.4.205.254:40081
I20260925 15:08:21.624233 15697 heartbeater.cc:464] Registering TS with master...
I20260925 15:08:21.624819 15697 heartbeater.cc:511] Master 127.4.205.254:40081 requested a full tablet report, sending...
I20260925 15:08:21.626026 15241 ts_manager.cc:194] Registered new tserver with Master: dbfce340dc3a4c47a9629e9ecf0c9d41 (127.4.205.195:41669)
I20260925 15:08:21.626734 15241 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:34053
I20260925 15:08:21.635856 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:08:21.650992 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.640151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.4.205.254:40081)
0925 15:08:21.640286 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.640289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.640793 (+ 504us) server_negotiation.cc:407] Connection header received
0925 15:08:21.640910 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.640914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.640961 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.641032 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:21.642382 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.643039 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.644196 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.644562 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.644647 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.646802 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:21.646806 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:21.646809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:21.646833 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:21.649010 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.649477 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.649498 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.649500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.649689 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:21.649988 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:21.649991 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:21.649992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:21.650242 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:21.650405 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.650662 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.650792 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:08:21.654341 15241 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40710:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:08:21.660743 15241 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:08:21.667500 15430 heartbeater.cc:503] Master 127.4.205.254:40081 was elected leader, sending a full tablet report...
I20260925 15:08:21.694686 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.672094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:39909 (local address 127.0.0.1:42156)
0925 15:08:21.685380 (+ 13286us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:21.685434 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:21.686168 (+ 734us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:21.687085 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:21.687093 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:21.687122 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:21.687599 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.687611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.690038 (+ 2427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.690043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.692236 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.692246 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.693754 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.693761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.693928 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.693988 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 15:08:21.694114 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:08:21.694276 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12990,"mutex_wait_us":29,"thread_start_us":14160,"threads_started":1}
I20260925 15:08:21.694902 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.672298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42156 (local address 127.4.205.193:39909)
0925 15:08:21.684548 (+ 12250us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.684561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.685569 (+ 1008us) server_negotiation.cc:407] Connection header received
0925 15:08:21.686341 (+ 772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.686349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.686614 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.686807 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:21.688379 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.689898 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.692384 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.693543 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.693687 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.693903 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.694165 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.694604 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12016,"thread_start_us":12264,"threads_started":1}
I20260925 15:08:21.695739 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.672398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41610 (local address 127.4.205.195:41669)
0925 15:08:21.686049 (+ 13651us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.686057 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.686087 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:08:21.686341 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.686349 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.686634 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.686807 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:21.688847 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.690266 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.693010 (+ 2744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.694254 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.694364 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.694565 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.695267 (+ 702us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.695425 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13488,"thread_start_us":13325,"threads_started":1}
I20260925 15:08:21.695755 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.672152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:41669 (local address 127.0.0.1:41610)
0925 15:08:21.685940 (+ 13788us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:21.685958 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:21.686113 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:21.687105 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:21.687109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:21.687138 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:21.688023 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.688039 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.690409 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.690415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.692843 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.692856 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.694699 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.694707 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.695093 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.695146 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:08:21.695230 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:08:21.695344 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13678,"mutex_wait_us":501,"spinlock_wait_cycles":4480,"thread_start_us":12769,"threads_started":1}
I20260925 15:08:21.695994 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.686836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58864 (local address 127.4.205.194:35967)
0925 15:08:21.687816 (+ 980us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:21.687825 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:21.688257 (+ 432us) server_negotiation.cc:407] Connection header received
0925 15:08:21.688459 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:21.688465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:21.688709 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:21.688897 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:21.689442 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.690765 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.693254 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.694460 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.694593 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.694772 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:21.695519 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 15:08:21.695675 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":765,"thread_start_us":611,"threads_started":1}
I20260925 15:08:21.696372 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:21.686580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35967 (local address 127.0.0.1:58864)
0925 15:08:21.688167 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:21.688184 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:21.688297 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:21.688899 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:21.688903 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:21.688933 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:21.689294 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.689306 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.690921 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.690928 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.693133 (+ 2205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:21.693142 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:21.694904 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:21.694910 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:21.695137 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:21.695165 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:08:21.695244 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:08:21.695354 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1426,"mutex_wait_us":38,"thread_start_us":1247,"threads_started":1}
I20260925 15:08:21.699436 15632 tablet_service.cc:1467] Processing CreateTablet for tablet f5e35201d42d4735b5f72aa134ad6061 (DEFAULT_TABLE table=test-table [id=49773705682c438c9eedf2e87a1a50c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:21.699422 15365 tablet_service.cc:1467] Processing CreateTablet for tablet f5e35201d42d4735b5f72aa134ad6061 (DEFAULT_TABLE table=test-table [id=49773705682c438c9eedf2e87a1a50c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:21.699635 15499 tablet_service.cc:1467] Processing CreateTablet for tablet f5e35201d42d4735b5f72aa134ad6061 (DEFAULT_TABLE table=test-table [id=49773705682c438c9eedf2e87a1a50c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:08:21.700707 15632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e35201d42d4735b5f72aa134ad6061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:21.700789 15365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e35201d42d4735b5f72aa134ad6061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:21.700793 15499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f5e35201d42d4735b5f72aa134ad6061. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:08:21.712597 15715 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Bootstrap starting.
I20260925 15:08:21.714813 15716 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Bootstrap starting.
I20260925 15:08:21.716279 15715 tablet_bootstrap.cc:654] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:21.717564 15715 log.cc:824] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:21.718210 15717 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Bootstrap starting.
I20260925 15:08:21.718444 15716 tablet_bootstrap.cc:654] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:21.719899 15716 log.cc:824] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:21.720922 15717 tablet_bootstrap.cc:654] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Neither blocks nor log segments found. Creating new log.
I20260925 15:08:21.722167 15717 log.cc:824] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:08:21.722983 15716 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: No bootstrap required, opened a new log
I20260925 15:08:21.723258 15716 ts_tablet_manager.cc:1397] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 15:08:21.723305 15715 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: No bootstrap required, opened a new log
I20260925 15:08:21.723543 15715 ts_tablet_manager.cc:1397] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 15:08:21.724853 15717 tablet_bootstrap.cc:492] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: No bootstrap required, opened a new log
I20260925 15:08:21.725077 15717 ts_tablet_manager.cc:1397] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:08:21.730489 15715 raft_consensus.cc:348] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.730829 15715 raft_consensus.cc:374] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:21.730929 15715 raft_consensus.cc:729] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbfce340dc3a4c47a9629e9ecf0c9d41, State: Initialized, Role: FOLLOWER
I20260925 15:08:21.731532 15715 consensus_queue.cc:260] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.732584 15697 heartbeater.cc:503] Master 127.4.205.254:40081 was elected leader, sending a full tablet report...
I20260925 15:08:21.733194 15715 ts_tablet_manager.cc:1428] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 15:08:21.733335 15716 raft_consensus.cc:348] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.733950 15716 raft_consensus.cc:374] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:21.734054 15716 raft_consensus.cc:729] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed1492fcbbb641a38e874ad7ece391aa, State: Initialized, Role: FOLLOWER
I20260925 15:08:21.734593 15716 consensus_queue.cc:260] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.735222 15717 raft_consensus.cc:348] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.735641 15717 raft_consensus.cc:374] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:08:21.735772 15717 raft_consensus.cc:729] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 270ca57733ed4c01a7be11da093f69d6, State: Initialized, Role: FOLLOWER
I20260925 15:08:21.736155 15716 ts_tablet_manager.cc:1428] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:08:21.736346 15717 consensus_queue.cc:260] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:21.737442 15564 heartbeater.cc:503] Master 127.4.205.254:40081 was elected leader, sending a full tablet report...
I20260925 15:08:21.737987 15717 ts_tablet_manager.cc:1428] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
W20260925 15:08:21.855010 15698 tablet.cc:2369] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:21.872587 15565 tablet.cc:2369] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:08:21.888652 15431 tablet.cc:2369] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:08:22.052706 15721 raft_consensus.cc:482] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:08:22.052990 15721 raft_consensus.cc:504] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:22.054582 15721 leader_election.cc:288] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed1492fcbbb641a38e874ad7ece391aa (127.4.205.193:39909), 270ca57733ed4c01a7be11da093f69d6 (127.4.205.194:35967)
I20260925 15:08:22.061242 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.054932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:33291 (local address 127.4.205.193:39909)
0925 15:08:22.055137 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.055142 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.055180 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:08:22.055258 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.055265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.055342 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.055459 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:22.056063 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.057065 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.059364 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.060502 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.060656 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.060727 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.060906 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.061040 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 15:08:22.061249 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.054847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:39909 (local address 127.4.205.195:33291)
0925 15:08:22.055029 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:22.055048 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:22.055158 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:22.055661 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:22.055666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:22.055692 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:22.055902 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:22.055930 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.057180 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.057183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:22.059178 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:22.059193 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.060627 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.060631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:22.060754 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.060776 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:08:22.060846 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:08:22.060930 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":44}
I20260925 15:08:22.062017 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.055410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:42347 (local address 127.4.205.194:35967)
0925 15:08:22.055625 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.055633 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.055850 (+ 217us) server_negotiation.cc:407] Connection header received
0925 15:08:22.056061 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.056065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.056140 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.056228 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:08:22.056698 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.057544 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.059746 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.060918 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.061044 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.061100 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.061536 (+ 436us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.061762 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260925 15:08:22.062110 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.055241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35967 (local address 127.4.205.195:42347)
0925 15:08:22.055759 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:08:22.055775 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:08:22.055886 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:08:22.056247 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:08:22.056250 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:08:22.056268 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:08:22.056561 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:22.056568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.057681 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.057686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:22.059591 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:08:22.059600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.061111 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.061117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:08:22.061278 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.061350 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 15:08:22.061511 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:08:22.061731 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":24,"thread_start_us":241,"threads_started":1}
I20260925 15:08:22.062348 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e35201d42d4735b5f72aa134ad6061" candidate_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1492fcbbb641a38e874ad7ece391aa" is_pre_election: true
I20260925 15:08:22.062834 15385 raft_consensus.cc:2454] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbfce340dc3a4c47a9629e9ecf0c9d41 in term 0.
I20260925 15:08:22.063014 15519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e35201d42d4735b5f72aa134ad6061" candidate_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270ca57733ed4c01a7be11da093f69d6" is_pre_election: true
I20260925 15:08:22.063496 15587 leader_election.cc:302] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbfce340dc3a4c47a9629e9ecf0c9d41, ed1492fcbbb641a38e874ad7ece391aa; no voters:
I20260925 15:08:22.063519 15519 raft_consensus.cc:2454] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbfce340dc3a4c47a9629e9ecf0c9d41 in term 0.
I20260925 15:08:22.063956 15721 raft_consensus.cc:2781] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:08:22.064137 15721 raft_consensus.cc:482] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:08:22.064255 15721 raft_consensus.cc:3037] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:22.066720 15721 raft_consensus.cc:504] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:22.067214 15721 leader_election.cc:288] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [CANDIDATE]: Term 1 election: Requested vote from peers ed1492fcbbb641a38e874ad7ece391aa (127.4.205.193:39909), 270ca57733ed4c01a7be11da093f69d6 (127.4.205.194:35967)
I20260925 15:08:22.067603 15385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e35201d42d4735b5f72aa134ad6061" candidate_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed1492fcbbb641a38e874ad7ece391aa"
I20260925 15:08:22.067736 15519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f5e35201d42d4735b5f72aa134ad6061" candidate_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "270ca57733ed4c01a7be11da093f69d6"
I20260925 15:08:22.067801 15385 raft_consensus.cc:3037] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:22.067934 15519 raft_consensus.cc:3037] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:08:22.069953 15385 raft_consensus.cc:2454] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfce340dc3a4c47a9629e9ecf0c9d41 in term 1.
I20260925 15:08:22.070047 15519 raft_consensus.cc:2454] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbfce340dc3a4c47a9629e9ecf0c9d41 in term 1.
I20260925 15:08:22.070271 15587 leader_election.cc:302] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dbfce340dc3a4c47a9629e9ecf0c9d41, ed1492fcbbb641a38e874ad7ece391aa; no voters:
I20260925 15:08:22.070521 15721 raft_consensus.cc:2781] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:08:22.071007 15721 raft_consensus.cc:686] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 1 LEADER]: Becoming Leader. State: Replica: dbfce340dc3a4c47a9629e9ecf0c9d41, State: Running, Role: LEADER
I20260925 15:08:22.071432 15721 consensus_queue.cc:237] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } }
I20260925 15:08:22.075825 15239 catalog_manager.cc:5495] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 reported cstate change: term changed from 0 to 1, leader changed from <none> to dbfce340dc3a4c47a9629e9ecf0c9d41 (127.4.205.195). New cstate: current_term: 1 leader_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbfce340dc3a4c47a9629e9ecf0c9d41" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41669 } health_report { overall_health: HEALTHY } } }
I20260925 15:08:22.149122 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.144256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41624 (local address 127.4.205.195:41669)
0925 15:08:22.144445 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.144452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.144546 (+ 94us) server_negotiation.cc:407] Connection header received
0925 15:08:22.144770 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.144774 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.144842 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.144994 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:08:22.145335 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.146065 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.147271 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.147629 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.147746 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.148206 (+ 460us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:08:22.148614 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:08:22.148768 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.148814 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.148882 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:08:22.156515 15385 raft_consensus.cc:1261] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 1 FOLLOWER]: Refusing update from remote peer dbfce340dc3a4c47a9629e9ecf0c9d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:22.156560 15519 raft_consensus.cc:1261] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 1 FOLLOWER]: Refusing update from remote peer dbfce340dc3a4c47a9629e9ecf0c9d41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:08:22.157408 15725 consensus_queue.cc:1035] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "270ca57733ed4c01a7be11da093f69d6" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:22.157835 15721 consensus_queue.cc:1035] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed1492fcbbb641a38e874ad7ece391aa" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 39909 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:08:22.167585 15728 mvcc.cc:205] Tried to move back new op lower bound from 7333269103223926784 to 7333269102888226816. Current Snapshot: MvccSnapshot[applied={T|T < 7333269103223926784}]
I20260925 15:08:22.170861 15729 mvcc.cc:205] Tried to move back new op lower bound from 7333269103223926784 to 7333269102888226816. Current Snapshot: MvccSnapshot[applied={T|T < 7333269103223926784}]
I20260925 15:08:22.180079 15241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:22.180408 15241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40710:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:08:22.183699 15241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c36a6ad9c2a43d99e55ecb4bfc546cd: Sending DeleteTablet for 3 replicas of tablet f5e35201d42d4735b5f72aa134ad6061
I20260925 15:08:22.184646 15365 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e35201d42d4735b5f72aa134ad6061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:22 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:42156
I20260925 15:08:22.184765 15499 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e35201d42d4735b5f72aa134ad6061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:22 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58864
I20260925 15:08:22.184985 15632 tablet_service.cc:1514] Processing DeleteTablet for tablet f5e35201d42d4735b5f72aa134ad6061 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:08:22 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:41610
I20260925 15:08:22.185760 15739 tablet_replica.cc:330] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: stopping tablet replica
I20260925 15:08:22.186102 15739 raft_consensus.cc:2229] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:22.186503 15739 raft_consensus.cc:2258] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:22.186630 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 15302
I20260925 15:08:22.188640 15739 ts_tablet_manager.cc:1905] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:22.195370 15738 tablet_replica.cc:330] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: stopping tablet replica
I20260925 15:08:22.195783 15738 raft_consensus.cc:2229] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:08:22.196261 15738 raft_consensus.cc:2258] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:22.194929 15739 ts_tablet_manager.cc:1918] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:22.196797 15739 log.cc:1196] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/f5e35201d42d4735b5f72aa134ad6061
I20260925 15:08:22.197365 15739 ts_tablet_manager.cc:1939] T f5e35201d42d4735b5f72aa134ad6061 P 270ca57733ed4c01a7be11da093f69d6: Deleting consensus metadata
I20260925 15:08:22.198391 15738 ts_tablet_manager.cc:1905] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:22.198939 15227 catalog_manager.cc:4841] TS 270ca57733ed4c01a7be11da093f69d6 (127.4.205.194:35967): tablet f5e35201d42d4735b5f72aa134ad6061 (table test-table [id=49773705682c438c9eedf2e87a1a50c2]) successfully deleted
I20260925 15:08:22.201576 15740 tablet_replica.cc:330] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: stopping tablet replica
I20260925 15:08:22.201999 15740 raft_consensus.cc:2229] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:08:22.202638 15740 raft_consensus.cc:2258] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:08:22.204433 15740 ts_tablet_manager.cc:1905] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:08:22.205651 15738 ts_tablet_manager.cc:1918] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:22.205852 15738 log.cc:1196] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/f5e35201d42d4735b5f72aa134ad6061
I20260925 15:08:22.206387 15738 ts_tablet_manager.cc:1939] T f5e35201d42d4735b5f72aa134ad6061 P ed1492fcbbb641a38e874ad7ece391aa: Deleting consensus metadata
I20260925 15:08:22.207800 15227 catalog_manager.cc:4841] TS ed1492fcbbb641a38e874ad7ece391aa (127.4.205.193:39909): tablet f5e35201d42d4735b5f72aa134ad6061 (table test-table [id=49773705682c438c9eedf2e87a1a50c2]) successfully deleted
I20260925 15:08:22.210456 15740 ts_tablet_manager.cc:1918] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:08:22.210630 15740 log.cc:1196] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/f5e35201d42d4735b5f72aa134ad6061
I20260925 15:08:22.211110 15740 ts_tablet_manager.cc:1939] T f5e35201d42d4735b5f72aa134ad6061 P dbfce340dc3a4c47a9629e9ecf0c9d41: Deleting consensus metadata
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:08:22.212426 15225 catalog_manager.cc:4841] TS dbfce340dc3a4c47a9629e9ecf0c9d41 (127.4.205.195:41669): tablet f5e35201d42d4735b5f72aa134ad6061 (table test-table [id=49773705682c438c9eedf2e87a1a50c2]) successfully deleted
I20260925 15:08:22.218827 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.187688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42158 (local address 127.4.205.193:39909)
0925 15:08:22.187847 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.187855 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.202120 (+ 14265us) server_negotiation.cc:407] Connection header received
0925 15:08:22.202519 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.202526 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.202600 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.202677 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:22.204355 (+ 1678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.205446 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.207616 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.208249 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.208345 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.213386 (+ 5041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:22.213399 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:22.213427 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:22.213464 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:22.216238 (+ 2774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.216863 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.216871 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.216874 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.216988 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.217358 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.217368 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.217375 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.217744 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:22.218042 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.218304 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.218512 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:08:22.219521 15425 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:22.303445 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 15302
I20260925 15:08:22.320243 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 15436
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:08:22.336586 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.321012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58876 (local address 127.4.205.194:35967)
0925 15:08:22.321174 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.321181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.321202 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:08:22.321456 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.321463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.321540 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.321665 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:22.322950 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.324068 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.325292 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.326011 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.326134 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.330187 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:22.330205 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:22.330225 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:22.330272 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:22.333809 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.334507 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.334516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.334520 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.334680 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.335019 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.335024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.335027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.335538 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:22.335812 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.336013 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.336221 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:08:22.337229 15559 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:22.425738 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 15436
I20260925 15:08:22.442801 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 15569
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15196](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:08:22.460250 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:08:22.443519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41854 (local address 127.4.205.195:41669)
0925 15:08:22.443692 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:08:22.443698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:08:22.443930 (+ 232us) server_negotiation.cc:407] Connection header received
0925 15:08:22.444134 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:08:22.444138 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:08:22.444220 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:08:22.444301 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:08:22.446005 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.446861 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.448762 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:08:22.449205 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:08:22.449296 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:08:22.454381 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:08:22.454392 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:08:22.454406 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:08:22.454441 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:08:22.457379 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.458180 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.458194 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.458196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.458338 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:08:22.458725 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:08:22.458728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:08:22.458730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:08:22.459080 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:08:22.459393 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:08:22.459671 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:08:22.459885 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:08:22.460906 15692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:22.575614 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 15569
I20260925 15:08:22.593492 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 15209
I20260925 15:08:22.594590 15271 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:08:22.696527 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 15209
2026-09-25T15:08:22Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4288 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15748](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15748](info): set up 2 sockets
Sep 25 15:08:22 dist-test-slave-64fx krb5kdc[15748](info): commencing operation
krb5kdc: starting...
W20260925 15:08:23.738412 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:08:23 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348903, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:08:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:08:23Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangeradmin_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangeradmin_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangerlookup_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangerlookup_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
I20260925 15:08:24.008432 4919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSXSwaN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/postgres -l logfile start
2026-09-25 15:08:26.303 UTC [15778] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:08:26.303 UTC [15778] LOG: listening on IPv4 address "127.4.205.212", port 35897
2026-09-25 15:08:26.306 UTC [15778] LOG: listening on Unix socket "/tmp/.s.PGSQL.35897"
2026-09-25 15:08:26.327 UTC [15781] LOG: database system was shut down at 2026-09-25 15:08:25 UTC
2026-09-25 15:08:26.330 UTC [15778] LOG: database system is ready to accept connections
I20260925 15:08:27.254134 4919 mini_postgres.cc:96] Postgres bound to 35897
2026-09-25 15:08:27.271 UTC [15789] FATAL: database "slave" does not exist
127.4.205.212:35897 - accepting connections
I20260925 15:08:27.271776 4919 mini_ranger.cc:161] Starting Ranger...
I20260925 15:08:27.295053 4919 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:08:27.726794 4919 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:08:27.726943 4919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin
2026-09-25 15:08:28,019 [I] DB FLAVOR :POSTGRES
2026-09-25 15:08:28,019 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:08:28,019 [I] Checking connection..
2026-09-25 15:08:28,019 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:08:28,285 [I] Checking connection passed.
2026-09-25 15:08:28,285 [I] --------- Verifying version history table ---------
2026-09-25 15:08:28,285 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:08:28,526 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:08:28,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:08:28,757 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:08:28,757 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:08:28,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:08:29.000 UTC [15853] WARNING: there is no transaction in progress
2026-09-25 15:08:29,006 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:08:29,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:08:29,225 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:08:29,225 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:08:29,225 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:08:29,459 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:08:29,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-64fx.c.gcp-upstream.internal','N') ;"
2026-09-25 15:08:29,943 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:08:30,189 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:08:30,190 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:08:30,431 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:08:30,431 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:08:30,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:08:30,861 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:08:30,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:08:31.780 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.796 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.812 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.828 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.846 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.866 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.930 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.937 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.951 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.961 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.971 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.984 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:31.994 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:32.000 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:32.007 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:32.014 UTC [15961] WARNING: there is no transaction in progress
2026-09-25 15:08:32,639 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:08:32,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:08:32,883 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:08:32,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:08:33,135 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:08:33,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:08:33,392 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:08:33,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:08:33,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:08:33,849 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-64fx.c.gcp-upstream.internal';"
2026-09-25 15:08:34,081 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:08:34,081 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:08:34,327 [I] DB_PATCHES have already been applied
I20260925 15:08:34.333108 4919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSXSwaN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:08:34.333190 4919 mini_ranger.cc:193] Using host: 127.4.205.212
I20260925 15:08:34.347261 4919 mini_ranger.cc:227] Ranger admin URL: http://127.4.205.212:39623
Sep 25, 2026 3:08:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:08:34 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:08:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:08:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:08:34 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:08:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:08:35 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:08:35.459009 4919 mini_ranger.cc:160] Time spent starting Ranger: real 8.187s user 0.003s sys 0.032s
Sep 25, 2026 3:08:54 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:08:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 3:08:54 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:08:54 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:08:55 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:08:56.113004 4919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangerkms_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/rangerkms_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.205.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.205.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:08:56 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:08:56.286986 4919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:08:56.304387 4919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:08:57.104754 4919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:08:57.104904 4919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms
I20260925 15:08:57.105034 4919 mini_ranger_kms.cc:227] Using postgres at 127.4.205.212:35897
2026-09-25 15:08:57,507 [I] DB FLAVOR :POSTGRES
2026-09-25 15:08:57,507 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:08:57,507 [I] Checking connection
2026-09-25 15:08:57,508 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:08:57,758 [I] connection success
2026-09-25 15:08:57,758 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:08:57,758 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:08:57,758 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:08:57,991 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:08:57,991 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:08:57,991 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:08:57,991 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:35897/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:08:58,240 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:08:58.721972 4919 mini_ranger_kms.cc:310] Created kms service
I20260925 15:08:58.883553 4919 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:08:58.926708 4919 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:08:58 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348936, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:08:59.299373 4919 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:08:59.299680 4919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:08:59.299746 4919 mini_ranger_kms.cc:240] Using host: 127.4.205.212
I20260925 15:08:59.319314 4919 mini_ranger_kms.cc:276] Ranger KMS PID: 16141
I20260925 15:08:59.319478 4919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.205.212:56869
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:09:00 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:09:01.565665 4919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.279s user 0.099s sys 0.315s
I20260925 15:09:01.565917 4919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:09:01.566143 4919 mini_ranger_kms.cc:401] 127.4.205.212:56869/kms/v1/keys
Sep 25 15:09:01 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:09:02 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:02 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348941, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:04 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:09:04.537969 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42083
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:42083
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:56869/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:04.863842 16192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:04.864135 16192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:04.864289 16192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:04.871220 16192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:04.871342 16192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:04.871444 16192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:04.871533 16192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:04.871613 16192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:04.880345 16192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.205.212:56869/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:42083
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:42083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:04.881517 16192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:04.882789 16192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:04.894438 16200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:04.894703 16198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:04.895037 16192 server_base.cc:1034] running on GCE node
W20260925 15:09:04.895100 16197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:04.895843 16192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:04.897099 16192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:04.898307 16192 hybrid_clock.cc:648] HybridClock initialized: now 1790348944898280 us; error 40 us; skew 500 ppm
I20260925 15:09:04.898627 16192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:04 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:04.906436 16192 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:09:04.907830 16192 webserver.cc:460] Webserver started at http://127.4.205.254:36295/ using document root <none> and password file <none>
I20260925 15:09:04.908409 16192 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:04.908543 16192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:04.908821 16192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:04 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:04.957072 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "533ee2b7eceb486499831783485c5ca0"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-64fx"
server_key: "edb36ecbf7f2d82d731fc7e295f27bbe"
server_key_iv: "ecda878c815d509fa96d248540e834f3"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:04.957966 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "533ee2b7eceb486499831783485c5ca0"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-64fx"
server_key: "edb36ecbf7f2d82d731fc7e295f27bbe"
server_key_iv: "ecda878c815d509fa96d248540e834f3"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:04.964007 16192 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 15:09:04.979538 16207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:04.981101 16192 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 15:09:04.981295 16192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "533ee2b7eceb486499831783485c5ca0"
format_stamp: "Formatted at 2026-09-25 15:09:04 on dist-test-slave-64fx"
server_key: "edb36ecbf7f2d82d731fc7e295f27bbe"
server_key_iv: "ecda878c815d509fa96d248540e834f3"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:04.981635 16192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:05.004628 16192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:05.008118 16192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:05.008414 16192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:05.026594 16192 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:42083
I20260925 15:09:05.026624 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:42083 every 8 connection(s)
I20260925 15:09:05.027877 16192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:05.031674 16259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:05.033520 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 16192
I20260925 15:09:05.033731 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:09:05.034471 4919 external_mini_cluster.cc:1428] Setting key c79944e1ddd8f2075935edc8bfd85194
I20260925 15:09:05.045465 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: Bootstrap starting.
I20260925 15:09:05.048712 16259 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: Neither blocks nor log segments found. Creating new log.
Sep 25 15:09:05 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:05.049726 16259 log.cc:824] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:05.052335 16259 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: No bootstrap required, opened a new log
I20260925 15:09:05.056906 16262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:05.036644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40272 (local address 127.4.205.254:42083)
0925 15:09:05.037418 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:05.037437 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:05.037484 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:09:05.039920 (+ 2436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:05.039946 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:05.040391 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:05.040803 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:05.041972 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.044426 (+ 2454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.045835 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:05.046812 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:05.046989 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:05.050540 (+ 3551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:05.050553 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:05.050573 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:05.050631 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:05.053392 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:05.054132 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:05.054137 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:05.054142 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:05.054278 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:05.054650 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:05.054653 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:05.054654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:05.055066 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:05.055248 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:05.055649 (+ 401us) server_negotiation.cc:299] Negotiation successful
0925 15:09:05.055953 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":326,"threads_started":1}
I20260925 15:09:05.060959 16259 raft_consensus.cc:348] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } }
I20260925 15:09:05.061273 16259 raft_consensus.cc:374] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:05.061421 16259 raft_consensus.cc:729] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533ee2b7eceb486499831783485c5ca0, State: Initialized, Role: FOLLOWER
I20260925 15:09:05.062060 16259 consensus_queue.cc:260] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } }
I20260925 15:09:05.062287 16259 raft_consensus.cc:388] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:05.062413 16259 raft_consensus.cc:482] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:05.062593 16259 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:05.064802 16259 raft_consensus.cc:504] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } }
I20260925 15:09:05.065258 16259 leader_election.cc:302] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 533ee2b7eceb486499831783485c5ca0; no voters:
I20260925 15:09:05.065780 16259 leader_election.cc:288] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:05.065908 16264 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:05.066617 16264 raft_consensus.cc:686] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [term 1 LEADER]: Becoming Leader. State: Replica: 533ee2b7eceb486499831783485c5ca0, State: Running, Role: LEADER
I20260925 15:09:05.067215 16264 consensus_queue.cc:237] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } }
I20260925 15:09:05.067690 16259 sys_catalog.cc:564] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:05.070299 16265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "533ee2b7eceb486499831783485c5ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } } }
I20260925 15:09:05.070735 16265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:05.071358 16266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 533ee2b7eceb486499831783485c5ca0. Latest consensus state: current_term: 1 leader_uuid: "533ee2b7eceb486499831783485c5ca0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "533ee2b7eceb486499831783485c5ca0" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 42083 } } }
I20260925 15:09:05.071622 16266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:05.074887 16272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:05.075070 16259 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:09:05.087286 16272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:05.098307 16272 catalog_manager.cc:1347] Generated new cluster ID: e7be6386489e452fa74b1e4f814520d8
I20260925 15:09:05.098487 16272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:05.106791 16272 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:05.108073 16272 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:05.118009 16272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: Generated new TSK 0
I20260925 15:09:05.118778 16272 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:09:05.942549 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:56869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:09:06.350440 16304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:06.350785 16304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:06.350986 16304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:06.361598 16304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:06.361743 16304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:06.361939 16304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:09:06.376572 16304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.205.212:56869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:06.378252 16304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:06.379787 16304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:06.401822 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:06.403081 16315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:06.404325 16317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:06.406925 16304 server_base.cc:1034] running on GCE node
I20260925 15:09:06.407598 16304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:06.408699 16304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:06.409950 16304 hybrid_clock.cc:648] HybridClock initialized: now 1790348946409883 us; error 54 us; skew 500 ppm
I20260925 15:09:06.410324 16304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:06.413942 16304 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:09:06.418725 16304 webserver.cc:460] Webserver started at http://127.4.205.193:33913/ using document root <none> and password file <none>
I20260925 15:09:06.419361 16304 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:06.419518 16304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:06.419817 16304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:06.440261 16304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "9f9111c539684b88b40545c0b0734268"
format_stamp: "Formatted at 2026-09-25 15:09:06 on dist-test-slave-64fx"
server_key: "ecce0f33daafa5a63c1c182c4f2317c9"
server_key_iv: "b721f7dd19ac2638856496b5f4b47abf"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:06.441720 16304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "9f9111c539684b88b40545c0b0734268"
format_stamp: "Formatted at 2026-09-25 15:09:06 on dist-test-slave-64fx"
server_key: "ecce0f33daafa5a63c1c182c4f2317c9"
server_key_iv: "b721f7dd19ac2638856496b5f4b47abf"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:06.458969 16304 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.010s sys 0.005s
I20260925 15:09:06.481913 16324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:06.483904 16304 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.006s sys 0.001s
I20260925 15:09:06.484144 16304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "9f9111c539684b88b40545c0b0734268"
format_stamp: "Formatted at 2026-09-25 15:09:06 on dist-test-slave-64fx"
server_key: "ecce0f33daafa5a63c1c182c4f2317c9"
server_key_iv: "b721f7dd19ac2638856496b5f4b47abf"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:06.484475 16304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:06.548933 16304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:06.553800 16304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:06.554088 16304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:06.558375 16304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:06.562343 16304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:06.562475 16304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:06.562695 16304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:06.562767 16304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:06.661954 16304 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:43927
I20260925 15:09:06.661969 16436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:43927 every 8 connection(s)
I20260925 15:09:06.664104 16304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:06.671463 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 16304
I20260925 15:09:06.671711 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:06.672693 4919 external_mini_cluster.cc:1428] Setting key c6e42519f0858f8c1636320665093de3
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
I20260925 15:09:06.724978 16440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:06.677899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:60601 (local address 127.4.205.254:42083)
0925 15:09:06.699326 (+ 21427us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:06.699332 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:06.700541 (+ 1209us) server_negotiation.cc:407] Connection header received
0925 15:09:06.705894 (+ 5353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:06.705901 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:06.705984 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:06.706060 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:06.709707 (+ 3647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:06.710818 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:06.713204 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:06.713862 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:06.713953 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:06.718117 (+ 4164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:06.718123 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:06.718127 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:06.718160 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:06.721611 (+ 3451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:06.722440 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:06.722445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:06.722449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:06.722602 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:06.723141 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:06.723146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:06.723149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:06.723446 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:06.723559 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:06.724386 (+ 827us) server_negotiation.cc:299] Negotiation successful
0925 15:09:06.724606 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21293,"thread_start_us":21436,"threads_started":1}
I20260925 15:09:06.727052 16439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:06.677280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42083 (local address 127.4.205.193:60601)
0925 15:09:06.700042 (+ 22762us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:06.700111 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:06.702876 (+ 2765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:06.708015 (+ 5139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:06.708027 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:06.708621 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:06.709370 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:06.709393 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:06.711006 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:06.711011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:06.713039 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:06.713054 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:06.714275 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:06.714281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:06.714528 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:06.715295 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:06.715323 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:06.717900 (+ 2577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:06.721758 (+ 3858us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:06.721767 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:06.721783 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:06.722311 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:06.722735 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:06.722740 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:06.722743 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:06.723014 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:06.723640 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:06.723647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:06.724011 (+ 364us) client_negotiation.cc:769] Sending connection context
0925 15:09:06.725247 (+ 1236us) client_negotiation.cc:240] Negotiation successful
0925 15:09:06.725756 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22410,"thread_start_us":27028,"threads_started":1}
I20260925 15:09:06.728816 16437 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42083
I20260925 15:09:06.729112 16437 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:06.729933 16437 heartbeater.cc:511] Master 127.4.205.254:42083 requested a full tablet report, sending...
I20260925 15:09:06.732214 16224 ts_manager.cc:194] Registered new tserver with Master: 9f9111c539684b88b40545c0b0734268 (127.4.205.193:43927)
I20260925 15:09:06.734212 16224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:60601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:09:06.752728 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:56869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:06 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348946, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
W20260925 15:09:07.229125 16443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:07.229573 16443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:07.229774 16443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:07.240960 16443 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:07.241104 16443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:07.241374 16443 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:09:07.251606 16443 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.205.212:56869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:07.252758 16443 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:07.254014 16443 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:07.267587 16458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.267645 16457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.268066 16460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:07.268471 16443 server_base.cc:1034] running on GCE node
I20260925 15:09:07.269022 16443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:07.270015 16443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:07.271214 16443 hybrid_clock.cc:648] HybridClock initialized: now 1790348947271165 us; error 43 us; skew 500 ppm
I20260925 15:09:07.271502 16443 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:07.274727 16443 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:09:07.276392 16443 webserver.cc:460] Webserver started at http://127.4.205.194:44307/ using document root <none> and password file <none>
I20260925 15:09:07.277868 16443 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:07.278004 16443 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:07.278365 16443 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:07.296008 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "9d0e025feb9146ecb450a856d188427b"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "2140e0ee8e202ce8d25e9705607bdc56"
server_key_iv: "198c2863bf09ae4bed06d15bb3697e6f"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.297020 16443 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "9d0e025feb9146ecb450a856d188427b"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "2140e0ee8e202ce8d25e9705607bdc56"
server_key_iv: "198c2863bf09ae4bed06d15bb3697e6f"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.303340 16443 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260925 15:09:07.319038 16467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.320540 16443 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:07.320719 16443 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "9d0e025feb9146ecb450a856d188427b"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "2140e0ee8e202ce8d25e9705607bdc56"
server_key_iv: "198c2863bf09ae4bed06d15bb3697e6f"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.320935 16443 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:07.337198 16443 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:07.341316 16443 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:07.341679 16443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:07.342562 16443 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:07.343858 16443 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:07.343954 16443 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.344102 16443 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:07.344174 16443 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.376503 16443 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:35601
I20260925 15:09:07.376538 16579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:35601 every 8 connection(s)
I20260925 15:09:07.377810 16443 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:07.388101 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 16443
I20260925 15:09:07.388278 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:07.388713 4919 external_mini_cluster.cc:1428] Setting key 0b6acac4a40a06c2f874bd2f4a51f67c
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:07.396601 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.379907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:40487 (local address 127.4.205.254:42083)
0925 15:09:07.380345 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.380350 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.381138 (+ 788us) server_negotiation.cc:407] Connection header received
0925 15:09:07.383312 (+ 2174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.383317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.383396 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.383488 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:07.384779 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.385547 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.387092 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.387467 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.387553 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.391098 (+ 3545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:07.391103 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:07.391106 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:07.391135 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:07.393647 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.394489 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.394495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.394498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.394614 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.395047 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.395050 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.395052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.395309 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:07.395410 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.396212 (+ 802us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.396359 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":205,"threads_started":1}
I20260925 15:09:07.398022 16583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.380373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42083 (local address 127.4.205.194:40487)
0925 15:09:07.380996 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:07.381034 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:07.383067 (+ 2033us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:07.383684 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:07.383692 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:07.384087 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:07.384642 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.384657 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.385711 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.385715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.386974 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.386992 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.387625 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.387629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.387805 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.388460 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:07.388479 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:07.390847 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:07.393770 (+ 2923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:07.393781 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:07.393797 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:07.394379 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:07.394717 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:07.394724 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:07.394731 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:07.394947 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:07.395439 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:07.395450 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:07.395842 (+ 392us) client_negotiation.cc:769] Sending connection context
0925 15:09:07.396127 (+ 285us) client_negotiation.cc:240] Negotiation successful
0925 15:09:07.396654 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":247,"threads_started":1}
I20260925 15:09:07.399361 16580 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42083
I20260925 15:09:07.399657 16580 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:07.400273 16580 heartbeater.cc:511] Master 127.4.205.254:42083 requested a full tablet report, sending...
I20260925 15:09:07.401710 16224 ts_manager.cc:194] Registered new tserver with Master: 9d0e025feb9146ecb450a856d188427b (127.4.205.194:35601)
I20260925 15:09:07.402937 16224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:40487
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:09:07.438371 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:56869/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:07.737797 16437 heartbeater.cc:503] Master 127.4.205.254:42083 was elected leader, sending a full tablet report...
W20260925 15:09:07.744338 16586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:07.744589 16586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:07.744750 16586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:07.751279 16586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:07.751401 16586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:07.751577 16586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:09:07.760335 16586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.205.212:56869/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:42083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:07.761418 16586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:07.762633 16586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:07.775180 16591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.775223 16592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:07.775245 16594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:07.777894 16586 server_base.cc:1034] running on GCE node
I20260925 15:09:07.778450 16586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:07.779379 16586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:07.780603 16586 hybrid_clock.cc:648] HybridClock initialized: now 1790348947780571 us; error 48 us; skew 500 ppm
I20260925 15:09:07.781242 16586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:07.784215 16586 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:09:07.785732 16586 webserver.cc:460] Webserver started at http://127.4.205.195:40921/ using document root <none> and password file <none>
I20260925 15:09:07.786396 16586 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:07.786528 16586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:07.786806 16586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:07.799330 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "e72bbfcb7608496483057d2e3835dabc"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "9d829c7c1795f665d40a48783f7994d3"
server_key_iv: "34e9dc50232c52d6429188401fe0b5fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.800112 16586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "e72bbfcb7608496483057d2e3835dabc"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "9d829c7c1795f665d40a48783f7994d3"
server_key_iv: "34e9dc50232c52d6429188401fe0b5fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.805150 16586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:09:07.818236 16601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.819558 16586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 15:09:07.819725 16586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "e72bbfcb7608496483057d2e3835dabc"
format_stamp: "Formatted at 2026-09-25 15:09:07 on dist-test-slave-64fx"
server_key: "9d829c7c1795f665d40a48783f7994d3"
server_key_iv: "34e9dc50232c52d6429188401fe0b5fd"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:07.819928 16586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:07.836717 16586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:07.840267 16586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:07.840531 16586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:07.841434 16586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:07.843101 16586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:07.843197 16586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.843345 16586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:07.843410 16586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:07.876073 16586 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:41289
I20260925 15:09:07.876104 16713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:41289 every 8 connection(s)
I20260925 15:09:07.877332 16586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:07.886359 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 16586
I20260925 15:09:07.886535 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:07.887055 4919 external_mini_cluster.cc:1428] Setting key b7a8b6563dbfdc4ffe2062521553bef9
Sep 25 15:09:07 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348947, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:07.895994 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.879717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:37545 (local address 127.4.205.254:42083)
0925 15:09:07.879895 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.879900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.881016 (+ 1116us) server_negotiation.cc:407] Connection header received
0925 15:09:07.882712 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.882716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.882794 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.882897 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:07.884141 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.884935 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.886862 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.887308 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.887430 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.891117 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:07.891122 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:07.891125 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:07.891153 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:07.893277 (+ 2124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.893983 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.893988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.893990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.894108 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.894502 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.894505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.894507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.894750 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:07.894911 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.895476 (+ 565us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.895678 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:09:07.896878 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.880228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:42083 (local address 127.4.205.195:37545)
0925 15:09:07.880848 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:07.880908 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:07.882480 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:07.883109 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:07.883116 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:07.883507 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:07.884012 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.884027 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.885100 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.885104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.886715 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.886737 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.887536 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.887544 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.887731 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.888346 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:07.888371 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:07.890944 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:07.893398 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:07.893404 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:07.893413 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:07.893865 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:07.894196 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:07.894199 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:07.894201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:07.894337 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:07.894879 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:07.894886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:07.895148 (+ 262us) client_negotiation.cc:769] Sending connection context
0925 15:09:07.895375 (+ 227us) client_negotiation.cc:240] Negotiation successful
0925 15:09:07.895799 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":244,"threads_started":1}
I20260925 15:09:07.898000 16714 heartbeater.cc:347] Connected to a master server at 127.4.205.254:42083
I20260925 15:09:07.898253 16714 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:07.898872 16714 heartbeater.cc:511] Master 127.4.205.254:42083 requested a full tablet report, sending...
I20260925 15:09:07.900125 16224 ts_manager.cc:194] Registered new tserver with Master: e72bbfcb7608496483057d2e3835dabc (127.4.205.195:41289)
I20260925 15:09:07.901132 16224 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:37545
I20260925 15:09:07.902067 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:07.917841 16582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.906218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.4.205.254:42083)
0925 15:09:07.906409 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.906418 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.906788 (+ 370us) server_negotiation.cc:407] Connection header received
0925 15:09:07.906919 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.906922 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.906984 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.907056 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:07.908332 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.909068 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.910736 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.911103 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.911183 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.913283 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:07.913287 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:07.913290 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:07.913314 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:07.915761 (+ 2447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.916293 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.916297 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.916299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.916408 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:07.916690 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:07.916693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:07.916695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:07.916913 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:07.917098 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.917337 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.917477 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:09:07.921378 16223 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:07.928767 16223 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:07.958123 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.939889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35601 (local address 127.0.0.1:37208)
0925 15:09:07.940852 (+ 963us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:07.940890 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:07.941037 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:07.942407 (+ 1370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:07.942416 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:07.942446 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:07.942940 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.942957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.948295 (+ 5338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.948303 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.953076 (+ 4773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.953095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.957015 (+ 3920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.957027 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.957304 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.957398 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 15:09:07.957546 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:09:07.957789 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":628,"threads_started":1}
I20260925 15:09:07.958566 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.940039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37208 (local address 127.4.205.194:35601)
0925 15:09:07.941518 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.941530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.941577 (+ 47us) server_negotiation.cc:407] Connection header received
0925 15:09:07.941678 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.941687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.941946 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.942122 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:07.944327 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.948113 (+ 3786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.953296 (+ 5183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.955291 (+ 1995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.955473 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.955726 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.958085 (+ 2359us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.958265 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1287,"thread_start_us":1169,"threads_started":1}
I20260925 15:09:07.962205 16515 tablet_service.cc:1467] Processing CreateTablet for tablet 38af74549730472a97f2d230f821db36 (DEFAULT_TABLE table=test-table [id=f55977587cb64e1080a37686069fe132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:07.963615 16515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38af74549730472a97f2d230f821db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:07.966467 16728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.941249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:43927 (local address 127.0.0.1:47798)
0925 15:09:07.941818 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:07.941838 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:07.941947 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:07.958763 (+ 16816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:07.958768 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:07.958793 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:07.959105 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.959116 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.961286 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.961290 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.963474 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.963487 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.965596 (+ 2109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.965602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.965777 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.965906 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 15:09:07.965980 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:09:07.966101 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"mutex_wait_us":128,"thread_start_us":355,"threads_started":1}
I20260925 15:09:07.966622 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.941753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.4.205.193:43927)
0925 15:09:07.956981 (+ 15228us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.956994 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.957029 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:09:07.957146 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.957157 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.957466 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.959288 (+ 1822us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:07.959528 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.961154 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.963631 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.965275 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.965849 (+ 574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.966052 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.966112 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.966285 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15005,"thread_start_us":16724,"threads_started":1}
I20260925 15:09:07.966950 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.941928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:41289 (local address 127.0.0.1:54158)
0925 15:09:07.958619 (+ 16691us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:07.958637 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:07.958763 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:07.959292 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:07.959296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:07.959329 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:07.959690 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.959702 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.961502 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.961508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.963430 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:07.963441 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.965551 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.965576 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:07.965803 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.965898 (+ 95us) client_negotiation.cc:769] Sending connection context
0925 15:09:07.965984 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:09:07.966110 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16574,"mutex_wait_us":96,"thread_start_us":16431,"threads_started":1}
I20260925 15:09:07.967339 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:07.942442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54158 (local address 127.4.205.195:41289)
0925 15:09:07.958188 (+ 15746us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:07.958199 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:07.958856 (+ 657us) server_negotiation.cc:407] Connection header received
0925 15:09:07.958947 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:07.958952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:07.959103 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:07.959316 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:07.959878 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.961374 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.963564 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:07.965315 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:07.965427 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:07.965645 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:07.966270 (+ 625us) server_negotiation.cc:299] Negotiation successful
0925 15:09:07.966435 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15546,"thread_start_us":15372,"threads_started":1}
I20260925 15:09:07.970222 16372 tablet_service.cc:1467] Processing CreateTablet for tablet 38af74549730472a97f2d230f821db36 (DEFAULT_TABLE table=test-table [id=f55977587cb64e1080a37686069fe132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:07.971069 16649 tablet_service.cc:1467] Processing CreateTablet for tablet 38af74549730472a97f2d230f821db36 (DEFAULT_TABLE table=test-table [id=f55977587cb64e1080a37686069fe132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:07.972117 16372 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38af74549730472a97f2d230f821db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:07.972477 16649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38af74549730472a97f2d230f821db36. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:07.977233 16732 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Bootstrap starting.
I20260925 15:09:07.980516 16732 tablet_bootstrap.cc:654] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:07.981782 16732 log.cc:824] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:07.983865 16734 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Bootstrap starting.
I20260925 15:09:07.984728 16732 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: No bootstrap required, opened a new log
I20260925 15:09:07.985029 16732 ts_tablet_manager.cc:1397] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 15:09:07.988004 16734 tablet_bootstrap.cc:654] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:07.989439 16734 log.cc:824] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:07.991819 16735 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Bootstrap starting.
I20260925 15:09:07.993085 16732 raft_consensus.cc:348] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:07.993417 16732 raft_consensus.cc:374] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:07.993510 16732 raft_consensus.cc:729] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0e025feb9146ecb450a856d188427b, State: Initialized, Role: FOLLOWER
I20260925 15:09:07.994244 16732 consensus_queue.cc:260] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:07.995815 16735 tablet_bootstrap.cc:654] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:07.996896 16734 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: No bootstrap required, opened a new log
I20260925 15:09:07.996974 16735 log.cc:824] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:07.997171 16734 ts_tablet_manager.cc:1397] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260925 15:09:08.000458 16735 tablet_bootstrap.cc:492] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: No bootstrap required, opened a new log
I20260925 15:09:08.000684 16735 ts_tablet_manager.cc:1397] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 15:09:08.001148 16580 heartbeater.cc:503] Master 127.4.205.254:42083 was elected leader, sending a full tablet report...
I20260925 15:09:08.002916 16732 ts_tablet_manager.cc:1428] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Time spent starting tablet: real 0.018s user 0.007s sys 0.010s
I20260925 15:09:08.007745 16734 raft_consensus.cc:348] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.008142 16734 raft_consensus.cc:374] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:08.008234 16734 raft_consensus.cc:729] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9111c539684b88b40545c0b0734268, State: Initialized, Role: FOLLOWER
I20260925 15:09:08.008800 16734 consensus_queue.cc:260] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.010393 16734 ts_tablet_manager.cc:1428] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Time spent starting tablet: real 0.013s user 0.014s sys 0.001s
I20260925 15:09:08.011053 16735 raft_consensus.cc:348] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.011372 16735 raft_consensus.cc:374] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:08.011463 16735 raft_consensus.cc:729] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e72bbfcb7608496483057d2e3835dabc, State: Initialized, Role: FOLLOWER
I20260925 15:09:08.012003 16735 consensus_queue.cc:260] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.012991 16714 heartbeater.cc:503] Master 127.4.205.254:42083 was elected leader, sending a full tablet report...
I20260925 15:09:08.013423 16735 ts_tablet_manager.cc:1428] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
W20260925 15:09:08.128801 16715 tablet.cc:2369] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:09:08.129669 16581 tablet.cc:2369] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:08.163859 16740 raft_consensus.cc:482] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:08.164155 16740 raft_consensus.cc:504] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.165773 16740 leader_election.cc:288] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9d0e025feb9146ecb450a856d188427b (127.4.205.194:35601), 9f9111c539684b88b40545c0b0734268 (127.4.205.193:43927)
I20260925 15:09:08.171826 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.166163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:54491 (local address 127.4.205.194:35601)
0925 15:09:08.166357 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.166362 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.166447 (+ 85us) server_negotiation.cc:407] Connection header received
0925 15:09:08.166718 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.166725 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.166813 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.166914 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:08.167527 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.168243 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.170070 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.170834 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.171081 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.171154 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.171455 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.171587 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:09:08.171841 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.166169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:35601 (local address 127.4.205.195:54491)
0925 15:09:08.166353 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:08.166373 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:08.166530 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:08.167096 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:08.167102 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:08.167137 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:08.167346 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.167359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.168375 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.168389 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.169929 (+ 1540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.169939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.171074 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.171079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.171268 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.171291 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:09:08.171422 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:09:08.171540 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:09:08.172636 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.166825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:34213 (local address 127.4.205.193:43927)
0925 15:09:08.167035 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.167042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.167352 (+ 310us) server_negotiation.cc:407] Connection header received
0925 15:09:08.167553 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.167557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.167619 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.167709 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:08.168190 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.169077 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.170863 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.171653 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.171844 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.171913 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.172208 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.172350 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 15:09:08.172663 16741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.166556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:43927 (local address 127.4.205.195:34213)
0925 15:09:08.167220 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:08.167236 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:08.167407 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:08.167728 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:08.167732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:08.167752 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:08.168049 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.168056 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.169209 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.169215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.170674 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:08.170686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.171849 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.171856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:08.172044 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.172086 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 15:09:08.172173 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:09:08.172284 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":382,"threads_started":1}
I20260925 15:09:08.173348 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38af74549730472a97f2d230f821db36" candidate_uuid: "e72bbfcb7608496483057d2e3835dabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0e025feb9146ecb450a856d188427b" is_pre_election: true
I20260925 15:09:08.173923 16535 raft_consensus.cc:2454] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72bbfcb7608496483057d2e3835dabc in term 0.
I20260925 15:09:08.174160 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38af74549730472a97f2d230f821db36" candidate_uuid: "e72bbfcb7608496483057d2e3835dabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9111c539684b88b40545c0b0734268" is_pre_election: true
I20260925 15:09:08.174599 16605 leader_election.cc:302] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0e025feb9146ecb450a856d188427b, e72bbfcb7608496483057d2e3835dabc; no voters:
I20260925 15:09:08.174674 16392 raft_consensus.cc:2454] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e72bbfcb7608496483057d2e3835dabc in term 0.
I20260925 15:09:08.174953 16740 raft_consensus.cc:2781] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:08.175094 16740 raft_consensus.cc:482] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:08.175203 16740 raft_consensus.cc:3037] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:08.177649 16740 raft_consensus.cc:504] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.178169 16740 leader_election.cc:288] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [CANDIDATE]: Term 1 election: Requested vote from peers 9d0e025feb9146ecb450a856d188427b (127.4.205.194:35601), 9f9111c539684b88b40545c0b0734268 (127.4.205.193:43927)
I20260925 15:09:08.178578 16535 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38af74549730472a97f2d230f821db36" candidate_uuid: "e72bbfcb7608496483057d2e3835dabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d0e025feb9146ecb450a856d188427b"
I20260925 15:09:08.178691 16392 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38af74549730472a97f2d230f821db36" candidate_uuid: "e72bbfcb7608496483057d2e3835dabc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f9111c539684b88b40545c0b0734268"
I20260925 15:09:08.178748 16535 raft_consensus.cc:3037] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:08.178902 16392 raft_consensus.cc:3037] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:08.180876 16535 raft_consensus.cc:2454] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72bbfcb7608496483057d2e3835dabc in term 1.
I20260925 15:09:08.181007 16392 raft_consensus.cc:2454] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e72bbfcb7608496483057d2e3835dabc in term 1.
I20260925 15:09:08.181226 16605 leader_election.cc:302] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0e025feb9146ecb450a856d188427b, e72bbfcb7608496483057d2e3835dabc; no voters:
I20260925 15:09:08.181535 16740 raft_consensus.cc:2781] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:08.182027 16740 raft_consensus.cc:686] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 1 LEADER]: Becoming Leader. State: Replica: e72bbfcb7608496483057d2e3835dabc, State: Running, Role: LEADER
I20260925 15:09:08.182536 16740 consensus_queue.cc:237] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } }
I20260925 15:09:08.187333 16223 catalog_manager.cc:5495] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc reported cstate change: term changed from 0 to 1, leader changed from <none> to e72bbfcb7608496483057d2e3835dabc (127.4.205.195). New cstate: current_term: 1 leader_uuid: "e72bbfcb7608496483057d2e3835dabc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e72bbfcb7608496483057d2e3835dabc" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 41289 } health_report { overall_health: HEALTHY } } }
W20260925 15:09:08.210927 16438 tablet.cc:2369] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:08.244529 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.239761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.4.205.195:41289)
0925 15:09:08.239965 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.239969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.239986 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:08.240176 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.240180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.240277 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.240404 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:08.240707 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.241377 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.242745 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.243100 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.243182 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.243568 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:08.244022 (+ 454us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:08.244123 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.244185 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.244241 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:09:08.252319 16535 raft_consensus.cc:1261] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 1 FOLLOWER]: Refusing update from remote peer e72bbfcb7608496483057d2e3835dabc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:08.252319 16392 raft_consensus.cc:1261] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 1 FOLLOWER]: Refusing update from remote peer e72bbfcb7608496483057d2e3835dabc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:08.253194 16742 consensus_queue.cc:1035] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d0e025feb9146ecb450a856d188427b" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 35601 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:08.253535 16740 consensus_queue.cc:1035] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f9111c539684b88b40545c0b0734268" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 43927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:08.268038 16746 mvcc.cc:205] Tried to move back new op lower bound from 7333269292031418368 to 7333269291759255552. Current Snapshot: MvccSnapshot[applied={T|T < 7333269292031418368}]
I20260925 15:09:08.277879 16223 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:08.278133 16223 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40288:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:08.281441 16223 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 533ee2b7eceb486499831783485c5ca0: Sending DeleteTablet for 3 replicas of tablet 38af74549730472a97f2d230f821db36
I20260925 15:09:08.282496 16515 tablet_service.cc:1514] Processing DeleteTablet for tablet 38af74549730472a97f2d230f821db36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:08 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:37208
I20260925 15:09:08.282717 16372 tablet_service.cc:1514] Processing DeleteTablet for tablet 38af74549730472a97f2d230f821db36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:08 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:47798
I20260925 15:09:08.283296 16649 tablet_service.cc:1514] Processing DeleteTablet for tablet 38af74549730472a97f2d230f821db36 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:08 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:54158
I20260925 15:09:08.284767 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 16304
I20260925 15:09:08.290659 16755 tablet_replica.cc:330] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: stopping tablet replica
I20260925 15:09:08.291225 16755 raft_consensus.cc:2229] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:08.291913 16755 raft_consensus.cc:2258] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:08.292781 16754 tablet_replica.cc:330] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: stopping tablet replica
I20260925 15:09:08.293220 16754 raft_consensus.cc:2229] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:08.293828 16754 raft_consensus.cc:2258] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:08.294183 16755 ts_tablet_manager.cc:1905] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:08.295804 16754 ts_tablet_manager.cc:1905] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:09:08 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:09:08.302641 16754 ts_tablet_manager.cc:1918] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:08.302781 16754 log.cc:1196] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/38af74549730472a97f2d230f821db36
I20260925 15:09:08.303261 16754 ts_tablet_manager.cc:1939] T 38af74549730472a97f2d230f821db36 P 9d0e025feb9146ecb450a856d188427b: Deleting consensus metadata
I20260925 15:09:08.304968 16211 catalog_manager.cc:4841] TS 9d0e025feb9146ecb450a856d188427b (127.4.205.194:35601): tablet 38af74549730472a97f2d230f821db36 (table test-table [id=f55977587cb64e1080a37686069fe132]) successfully deleted
I20260925 15:09:08.305835 16755 ts_tablet_manager.cc:1918] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:08.305920 16756 tablet_replica.cc:330] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: stopping tablet replica
I20260925 15:09:08.305984 16755 log.cc:1196] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/38af74549730472a97f2d230f821db36
I20260925 15:09:08.306308 16756 raft_consensus.cc:2229] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:08.306525 16755 ts_tablet_manager.cc:1939] T 38af74549730472a97f2d230f821db36 P 9f9111c539684b88b40545c0b0734268: Deleting consensus metadata
I20260925 15:09:08.306890 16756 raft_consensus.cc:2258] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:08.308506 16211 catalog_manager.cc:4841] TS 9f9111c539684b88b40545c0b0734268 (127.4.205.193:43927): tablet 38af74549730472a97f2d230f821db36 (table test-table [id=f55977587cb64e1080a37686069fe132]) successfully deleted
I20260925 15:09:08.309413 16756 ts_tablet_manager.cc:1905] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:08.311560 16729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.285868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47808 (local address 127.4.205.193:43927)
0925 15:09:08.286085 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.286093 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.290100 (+ 4007us) server_negotiation.cc:407] Connection header received
0925 15:09:08.290408 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.290416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.290523 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.290604 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.292751 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.293831 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.295668 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.296324 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.296427 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.302569 (+ 6142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.302587 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.302608 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.302659 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.307387 (+ 4728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.309643 (+ 2256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.309649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.309651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.309760 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.310129 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.310135 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.310139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.310447 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:08.310712 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.311014 (+ 302us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.311226 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:09:08.312119 16432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:08.316947 16756 ts_tablet_manager.cc:1918] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:08.317106 16756 log.cc:1196] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/38af74549730472a97f2d230f821db36
I20260925 15:09:08.317678 16756 ts_tablet_manager.cc:1939] T 38af74549730472a97f2d230f821db36 P e72bbfcb7608496483057d2e3835dabc: Deleting consensus metadata
I20260925 15:09:08.319145 16211 catalog_manager.cc:4841] TS e72bbfcb7608496483057d2e3835dabc (127.4.205.195:41289): tablet 38af74549730472a97f2d230f821db36 (table test-table [id=f55977587cb64e1080a37686069fe132]) successfully deleted
I20260925 15:09:08.405251 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 16304
I20260925 15:09:08.422832 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 16443
Sep 25 15:09:08 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:09:08.439565 16727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.423703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37218 (local address 127.4.205.194:35601)
0925 15:09:08.423874 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.423880 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.423900 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:09:08.423960 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.423965 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.424054 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.424155 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.425452 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.426653 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.427939 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.428548 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.428705 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.433015 (+ 4310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.433031 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.433051 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.433088 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.436763 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.437432 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.437442 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.437446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.437640 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.438006 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.438013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.438017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.438478 (+ 461us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:08.438753 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.438979 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.439182 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:09:08.440260 16575 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:08.544776 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 16443
I20260925 15:09:08.561648 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 16586
Sep 25 15:09:08 dist-test-slave-64fx krb5kdc[15748](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348944, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:09:08.580019 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:08.562464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54176 (local address 127.4.205.195:41289)
0925 15:09:08.562644 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:08.562648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:08.562666 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:09:08.562859 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:08.562863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:08.562930 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:08.563015 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:08.564790 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.565712 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.567556 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:08.568036 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:08.568145 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:08.573492 (+ 5347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:08.573505 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:08.573523 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:08.573701 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:08.577331 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.578127 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.578134 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.578136 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.578286 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:08.578679 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:08.578682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:08.578683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:08.578993 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:08.579202 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:08.579575 (+ 373us) server_negotiation.cc:299] Negotiation successful
0925 15:09:08.579713 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:09:08.580641 16709 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:08.695344 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 16586
I20260925 15:09:08.713876 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 16192
I20260925 15:09:08.715027 16254 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:08.822459 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 16192
I20260925 15:09:08.844118 4919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:09:09.180173 4919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:09:09.180421 4919 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:09:09.526079 4919 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:09:09.526 UTC [15778] LOG: received smart shutdown request
2026-09-25 15:09:09.529 UTC [15778] LOG: background worker "logical replication launcher" (PID 15787) exited with exit code 1
2026-09-25 15:09:09.530 UTC [15782] LOG: shutting down
2026-09-25 15:09:09.680 UTC [15778] LOG: database system is shut down
2026-09-25T15:09:09Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47108 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:09:09 dist-test-slave-64fx krb5kdc[16770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:09:09 dist-test-slave-64fx krb5kdc[16770](info): set up 2 sockets
Sep 25 15:09:09 dist-test-slave-64fx krb5kdc[16770](info): commencing operation
krb5kdc: starting...
W20260925 15:09:10.869876 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:09:11 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348951, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:09:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:09:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangeradmin_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangeradmin_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangerlookup_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangerlookup_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
I20260925 15:09:11.160306 4919 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskSXSwaN/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/postgres -l logfile start
2026-09-25 15:09:13.435 UTC [16800] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 15:09:13.435 UTC [16800] LOG: listening on IPv4 address "127.4.205.212", port 38859
2026-09-25 15:09:13.438 UTC [16800] LOG: listening on Unix socket "/tmp/.s.PGSQL.38859"
2026-09-25 15:09:13.460 UTC [16803] LOG: database system was shut down at 2026-09-25 15:09:12 UTC
2026-09-25 15:09:13.463 UTC [16800] LOG: database system is ready to accept connections
I20260925 15:09:14.393479 4919 mini_postgres.cc:96] Postgres bound to 38859
2026-09-25 15:09:14.413 UTC [16811] FATAL: database "slave" does not exist
127.4.205.212:38859 - accepting connections
I20260925 15:09:14.414315 4919 mini_ranger.cc:161] Starting Ranger...
I20260925 15:09:14.438544 4919 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 15:09:14.864176 4919 mini_ranger.cc:87] Created ranger Postgres database
I20260925 15:09:14.864289 4919 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin
2026-09-25 15:09:15,134 [I] DB FLAVOR :POSTGRES
2026-09-25 15:09:15,135 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:09:15,135 [I] Checking connection..
2026-09-25 15:09:15,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 15:09:15,358 [I] Checking connection passed.
2026-09-25 15:09:15,358 [I] --------- Verifying version history table ---------
2026-09-25 15:09:15,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:09:15,612 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:09:15,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:09:15,862 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 15:09:15,862 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 15:09:15,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 15:09:16.106 UTC [16875] WARNING: there is no transaction in progress
2026-09-25 15:09:16,112 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 15:09:16,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 15:09:16,360 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 15:09:16,361 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:09:16,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 15:09:16,609 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 15:09:16,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-64fx.c.gcp-upstream.internal','N') ;"
2026-09-25 15:09:17,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:09:17,293 [I] Table x_portal_user does not exist in database ranger
2026-09-25 15:09:17,293 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:09:17,516 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 15:09:17,516 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:09:17,738 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:09:17,980 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 15:09:17,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 15:09:18.893 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:18.908 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:18.922 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:18.936 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:18.950 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:18.966 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.016 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.022 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.036 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.045 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.055 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.068 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.077 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.082 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.088 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19.095 UTC [16983] WARNING: there is no transaction in progress
2026-09-25 15:09:19,662 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 15:09:19,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 15:09:19,876 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 15:09:19,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 15:09:20,125 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 15:09:20,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 15:09:20,376 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 15:09:20,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 15:09:20,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 15:09:20,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-64fx.c.gcp-upstream.internal';"
2026-09-25 15:09:21,104 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 15:09:21,105 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 15:09:21,354 [I] DB_PATCHES have already been applied
I20260925 15:09:21.359787 4919 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskSXSwaN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 15:09:21.359862 4919 mini_ranger.cc:193] Using host: 127.4.205.212
I20260925 15:09:21.373274 4919 mini_ranger.cc:227] Ranger admin URL: http://127.4.205.212:47617
Sep 25, 2026 3:09:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:09:21 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:09:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 3:09:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 3:09:21 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 3:09:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 3:09:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 15:09:22.491223 4919 mini_ranger.cc:160] Time spent starting Ranger: real 8.077s user 0.011s sys 0.023s
Sep 25, 2026 3:09:42 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:09:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 3:09:42 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:09:42 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:09:43 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:09:43.832192 4919 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.4.205.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.4.205.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangerkms_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.4.205.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/rangerkms_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.4.205.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.4.205.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.4.205.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/HTTP_127.4.205.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:09:43 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348983, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:09:43.982012 4919 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:09:44.003360 4919 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:09:44.816246 4919 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:09:44.816355 4919 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms
I20260925 15:09:44.816408 4919 mini_ranger_kms.cc:227] Using postgres at 127.4.205.212:38859
2026-09-25 15:09:45,148 [I] DB FLAVOR :POSTGRES
2026-09-25 15:09:45,148 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:09:45,148 [I] Checking connection
2026-09-25 15:09:45,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:09:45,362 [I] connection success
2026-09-25 15:09:45,362 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:09:45,363 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:09:45,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:09:45,596 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:09:45,597 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:09:45,597 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:09:45,597 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.4.205.212:38859/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:09:45,853 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:09:46.284708 4919 mini_ranger_kms.cc:310] Created kms service
I20260925 15:09:46.451778 4919 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:09:46.492542 4919 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:09:46 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348983, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:46.865669 4919 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:09:46.865971 4919 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms:/tmp/dist-test-taskSXSwaN/build/asan/bin/postgresql.jar:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskSXSwaN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:09:46.866034 4919 mini_ranger_kms.cc:240] Using host: 127.4.205.212
I20260925 15:09:46.879983 4919 mini_ranger_kms.cc:276] Ranger KMS PID: 17163
I20260925 15:09:46.880106 4919 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.4.205.212:50835
Sep 25, 2026 3:09:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:09:47 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:09:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:09:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:09:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:09:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:09:49.085436 4919 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.103s user 0.073s sys 0.273s
I20260925 15:09:49.085713 4919 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:09:49.085892 4919 mini_ranger_kms.cc:401] 127.4.205.212:50835/kms/v1/keys
Sep 25 15:09:49 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:09:49 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:49 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348989, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.4.205.212@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:52 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:09:52.071178 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:45099
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:45099
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:52.377267 17214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:52.377513 17214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:52.377662 17214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:52.384212 17214 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:52.384294 17214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:52.384333 17214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:52.384375 17214 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:52.384409 17214 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:52.393411 17214 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
--ranger_kms_url=127.4.205.212:50835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:45099
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:52.394594 17214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:52.396391 17214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:52.411451 17219 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:52.411381 17220 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:52.411429 17222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:52.411708 17214 server_base.cc:1034] running on GCE node
I20260925 15:09:52.412551 17214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:52.413800 17214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:52.415016 17214 hybrid_clock.cc:648] HybridClock initialized: now 1790348992414966 us; error 47 us; skew 500 ppm
I20260925 15:09:52.415328 17214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:52 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:52.418176 17214 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:09:52.419421 17214 webserver.cc:460] Webserver started at http://127.4.205.254:36381/ using document root <none> and password file <none>
I20260925 15:09:52.420009 17214 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:52.420137 17214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:52.420433 17214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:52 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:52.476784 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "5027fe8b02f140c6bd0ae6456f3093ef"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-64fx"
server_key: "849f867809bb37a49367db027e439575"
server_key_iv: "4628a76433640e3d492c6eb365de7a3d"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:52.477846 17214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "5027fe8b02f140c6bd0ae6456f3093ef"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-64fx"
server_key: "849f867809bb37a49367db027e439575"
server_key_iv: "4628a76433640e3d492c6eb365de7a3d"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:52.484516 17214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260925 15:09:52.500550 17229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:52.502241 17214 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:09:52.502408 17214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "5027fe8b02f140c6bd0ae6456f3093ef"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-64fx"
server_key: "849f867809bb37a49367db027e439575"
server_key_iv: "4628a76433640e3d492c6eb365de7a3d"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:52.502660 17214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:52.527794 17214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:52.531013 17214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:52.531245 17214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:52.548689 17214 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:45099
I20260925 15:09:52.548708 17280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:45099 every 8 connection(s)
I20260925 15:09:52.550120 17214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:52.551203 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17214
I20260925 15:09:52.551419 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:09:52.552067 4919 external_mini_cluster.cc:1428] Setting key aeb5ac5223911d8eb94df1285469bf5f
I20260925 15:09:52.554085 17281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:09:52 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:52.566776 17281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Bootstrap starting.
I20260925 15:09:52.569701 17281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:52.570734 17281 log.cc:824] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:52.573675 17281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: No bootstrap required, opened a new log
I20260925 15:09:52.574314 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:52.553952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59858 (local address 127.4.205.254:45099)
0925 15:09:52.554868 (+ 916us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:52.554882 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:52.554926 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:09:52.557051 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:52.557076 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:52.557559 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:52.558006 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:52.559048 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.561105 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.562410 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:52.563391 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:52.563559 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:52.567484 (+ 3925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:52.567491 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:52.567504 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:52.567534 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:52.570449 (+ 2915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.571232 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.571244 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.571254 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.571487 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:52.571841 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:52.571846 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:52.571850 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:52.572334 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:52.572551 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:52.572963 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 15:09:52.573269 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":479,"threads_started":1}
I20260925 15:09:52.581727 17281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:52.582060 17281 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:52.582176 17281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5027fe8b02f140c6bd0ae6456f3093ef, State: Initialized, Role: FOLLOWER
I20260925 15:09:52.582697 17281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:52.582906 17281 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:52.583047 17281 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:52.583204 17281 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:52.585170 17281 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:52.585680 17281 leader_election.cc:302] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5027fe8b02f140c6bd0ae6456f3093ef; no voters:
I20260925 15:09:52.586223 17281 leader_election.cc:288] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:09:52.586340 17286 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:52.587004 17286 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 LEADER]: Becoming Leader. State: Replica: 5027fe8b02f140c6bd0ae6456f3093ef, State: Running, Role: LEADER
I20260925 15:09:52.587522 17286 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:52.588289 17281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:52.590503 17287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } } }
I20260925 15:09:52.590435 17288 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5027fe8b02f140c6bd0ae6456f3093ef. Latest consensus state: current_term: 1 leader_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } } }
I20260925 15:09:52.590857 17287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:52.590891 17288 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:52.596184 17293 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:52.596526 17281 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:09:52.608148 17293 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:52.619354 17293 catalog_manager.cc:1347] Generated new cluster ID: 4550202c06784cbc89f3e5f9ff3d3888
I20260925 15:09:52.619462 17293 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:52.628906 17293 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:09:52.630259 17293 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:52.639597 17293 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Generated new TSK 0
I20260925 15:09:52.640336 17293 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:09:53.357512 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:53 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 15:09:53.760406 17326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:53.760825 17326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:53.760993 17326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:53.772044 17326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:53.772164 17326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:53.772351 17326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:09:53.787015 17326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:53.788539 17326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:53.790226 17326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:53.805328 17336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:53.806770 17337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:53.818918 17339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:53.819537 17326 server_base.cc:1034] running on GCE node
I20260925 15:09:53.820487 17326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:53.823350 17326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:53.824633 17326 hybrid_clock.cc:648] HybridClock initialized: now 1790348993824576 us; error 47 us; skew 500 ppm
I20260925 15:09:53.824987 17326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:53 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:53.834506 17326 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:09:53.836421 17326 webserver.cc:460] Webserver started at http://127.4.205.193:34735/ using document root <none> and password file <none>
I20260925 15:09:53.837311 17326 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:53.837436 17326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:53.837780 17326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:53 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:53.855831 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "66c93d5dd46d4839af68f68af0fc7bc3"
format_stamp: "Formatted at 2026-09-25 15:09:53 on dist-test-slave-64fx"
server_key: "a51bf6578d284386719962372128cbb9"
server_key_iv: "2ba30ff1348de9cd093793867a1be416"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:53.857009 17326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "66c93d5dd46d4839af68f68af0fc7bc3"
format_stamp: "Formatted at 2026-09-25 15:09:53 on dist-test-slave-64fx"
server_key: "a51bf6578d284386719962372128cbb9"
server_key_iv: "2ba30ff1348de9cd093793867a1be416"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:53.871635 17326 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.001s sys 0.010s
I20260925 15:09:53.900645 17346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:53.902529 17326 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.003s sys 0.006s
I20260925 15:09:53.902770 17326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "66c93d5dd46d4839af68f68af0fc7bc3"
format_stamp: "Formatted at 2026-09-25 15:09:53 on dist-test-slave-64fx"
server_key: "a51bf6578d284386719962372128cbb9"
server_key_iv: "2ba30ff1348de9cd093793867a1be416"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:53.903105 17326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:53.962634 17326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:53.967260 17326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:53.967576 17326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:53.968607 17326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:53.970242 17326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:53.970351 17326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:53.970475 17326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:53.970536 17326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:54.036238 17326 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:34377
I20260925 15:09:54.036279 17458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:34377 every 8 connection(s)
I20260925 15:09:54.038039 17326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:54.045524 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17326
I20260925 15:09:54.045763 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:09:54.046593 4919 external_mini_cluster.cc:1428] Setting key 8f31dc7da70269ac5bb3481d0b02e193
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:54.079366 17462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:54.056661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:59789 (local address 127.4.205.254:45099)
0925 15:09:54.060994 (+ 4333us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:54.061000 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:54.061018 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:09:54.061075 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:54.061080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:54.061168 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:54.061296 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:54.064004 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.065156 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.067561 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.068164 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.068286 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:54.072975 (+ 4689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:54.072982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:54.072985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:54.073020 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:54.076158 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:54.076963 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:54.076968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:54.076971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:54.077098 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:54.077521 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:54.077525 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:54.077528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:54.077884 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:54.078008 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:54.078832 (+ 824us) server_negotiation.cc:299] Negotiation successful
0925 15:09:54.079045 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4218,"thread_start_us":4130,"threads_started":1}
I20260925 15:09:54.081444 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:54.056097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.193:59789)
0925 15:09:54.057951 (+ 1854us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:54.058008 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:54.060353 (+ 2345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:54.061749 (+ 1396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:54.061772 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:54.062438 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:54.063261 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:54.063284 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.065352 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.065357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:54.067406 (+ 2049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:54.067420 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.068600 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.068607 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:54.068889 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:54.069625 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:54.069653 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:54.072720 (+ 3067us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:54.076326 (+ 3606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:54.076334 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:54.076348 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:54.076845 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:54.077215 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:54.077219 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:54.077222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:54.077414 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:54.078092 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:54.078099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:54.078458 (+ 359us) client_negotiation.cc:769] Sending connection context
0925 15:09:54.079644 (+ 1186us) client_negotiation.cc:240] Negotiation successful
0925 15:09:54.080144 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1547,"thread_start_us":1358,"threads_started":1}
I20260925 15:09:54.083424 17459 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:54.083771 17459 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:54.084666 17459 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:09:54.087083 17246 ts_manager.cc:194] Registered new tserver with Master: 66c93d5dd46d4839af68f68af0fc7bc3 (127.4.205.193:34377)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
I20260925 15:09:54.088943 17246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:59789
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:09:54.123198 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348993, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
W20260925 15:09:54.571882 17465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:54.572233 17465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:54.572440 17465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:54.583776 17465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:54.583963 17465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:54.584180 17465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:09:54.598744 17465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:54.600450 17465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:54.602244 17465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:54.635490 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:54.637343 17473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:54.641841 17465 server_base.cc:1034] running on GCE node
W20260925 15:09:54.642906 17476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:54.643702 17465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:54.649832 17465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:54.651132 17465 hybrid_clock.cc:648] HybridClock initialized: now 1790348994651027 us; error 93 us; skew 500 ppm
I20260925 15:09:54.651500 17465 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:54.655612 17465 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:09:54.657426 17465 webserver.cc:460] Webserver started at http://127.4.205.194:42341/ using document root <none> and password file <none>
I20260925 15:09:54.658299 17465 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:54.658423 17465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:54.658718 17465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:54.680431 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "3fcaad890ec04ee19ba2bcd2e9560f73"
format_stamp: "Formatted at 2026-09-25 15:09:54 on dist-test-slave-64fx"
server_key: "67e5b70f1323b9d4cc289ea86da5c572"
server_key_iv: "ee052aa58802cc289783db0d61e2f8aa"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:54.681519 17465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "3fcaad890ec04ee19ba2bcd2e9560f73"
format_stamp: "Formatted at 2026-09-25 15:09:54 on dist-test-slave-64fx"
server_key: "67e5b70f1323b9d4cc289ea86da5c572"
server_key_iv: "ee052aa58802cc289783db0d61e2f8aa"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:54.699486 17465 fs_manager.cc:691] Time spent creating directory manager: real 0.017s user 0.011s sys 0.004s
I20260925 15:09:54.720063 17489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:54.721938 17465 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:09:54.722170 17465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "3fcaad890ec04ee19ba2bcd2e9560f73"
format_stamp: "Formatted at 2026-09-25 15:09:54 on dist-test-slave-64fx"
server_key: "67e5b70f1323b9d4cc289ea86da5c572"
server_key_iv: "ee052aa58802cc289783db0d61e2f8aa"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:54.722468 17465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:54.738226 17465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:54.741901 17465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:54.742228 17465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:54.743180 17465 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:54.744544 17465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:54.744634 17465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:54.744807 17465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:54.744899 17465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:54.814601 17465 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:44345
I20260925 15:09:54.814620 17601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:44345 every 8 connection(s)
I20260925 15:09:54.816049 17465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:54.825671 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17465
I20260925 15:09:54.825837 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:09:54.826352 4919 external_mini_cluster.cc:1428] Setting key 4dcf9d25390993fee602b482478fef58
Sep 25 15:09:54 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348994, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:54.842299 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:54.820787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:45239 (local address 127.4.205.254:45099)
0925 15:09:54.824255 (+ 3468us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:54.824260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:54.824283 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:09:54.825060 (+ 777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:54.825066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:54.825158 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:54.825277 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:54.828870 (+ 3593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.829699 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.831232 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.831631 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.831778 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:54.835162 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:54.835168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:54.835171 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:54.835201 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:54.838541 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:54.839492 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:54.839500 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:54.839506 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:54.839668 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:54.840184 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:54.840189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:54.840192 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:54.840560 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:54.840699 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:54.841693 (+ 994us) server_negotiation.cc:299] Negotiation successful
0925 15:09:54.841920 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3348,"thread_start_us":3246,"threads_started":1}
I20260925 15:09:54.843587 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:54.820076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.194:45239)
0925 15:09:54.821913 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:54.821967 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:54.824772 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:54.827120 (+ 2348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:54.827134 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:54.827809 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:54.828662 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:54.828709 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.829836 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.829842 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:54.831077 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:54.831095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:54.831857 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:54.831864 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:54.832087 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:54.832766 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:54.832788 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:54.834954 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:54.838689 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:54.838698 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:54.838714 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:54.839339 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:54.839792 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:54.839796 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:54.839800 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:54.840053 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:54.840704 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:54.840714 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:54.841180 (+ 466us) client_negotiation.cc:769] Sending connection context
0925 15:09:54.841515 (+ 335us) client_negotiation.cc:240] Negotiation successful
0925 15:09:54.842120 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1512,"thread_start_us":1252,"threads_started":1}
I20260925 15:09:54.845146 17602 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:54.845463 17602 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:54.846308 17602 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:54.847990 17246 ts_manager.cc:194] Registered new tserver with Master: 3fcaad890ec04ee19ba2bcd2e9560f73 (127.4.205.194:44345)
I20260925 15:09:54.849231 17246 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:45239
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:09:54.881052 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:55.092698 17459 heartbeater.cc:503] Master 127.4.205.254:45099 was elected leader, sending a full tablet report...
W20260925 15:09:55.188113 17608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:55.188416 17608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:55.188591 17608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:55.195603 17608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:55.195742 17608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:55.195905 17608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:09:55.205250 17608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:55.206455 17608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:55.207769 17608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:55.219873 17613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:55.220022 17616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:55.220047 17614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:55.220817 17608 server_base.cc:1034] running on GCE node
I20260925 15:09:55.221314 17608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:55.222283 17608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:55.223496 17608 hybrid_clock.cc:648] HybridClock initialized: now 1790348995223461 us; error 62 us; skew 500 ppm
I20260925 15:09:55.223783 17608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348995, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:55.226814 17608 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:09:55.228174 17608 webserver.cc:460] Webserver started at http://127.4.205.195:43067/ using document root <none> and password file <none>
I20260925 15:09:55.228775 17608 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:55.228904 17608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:09:55.229249 17608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348995, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:55.244164 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "832ab9ba84424d448fc01e9f3e014a63"
format_stamp: "Formatted at 2026-09-25 15:09:55 on dist-test-slave-64fx"
server_key: "e05deb248fe46ac0eeaa7c8c4f101956"
server_key_iv: "14dcf0501a037fbbfe96a16951a3d2c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:55.244971 17608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "832ab9ba84424d448fc01e9f3e014a63"
format_stamp: "Formatted at 2026-09-25 15:09:55 on dist-test-slave-64fx"
server_key: "e05deb248fe46ac0eeaa7c8c4f101956"
server_key_iv: "14dcf0501a037fbbfe96a16951a3d2c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:55.249897 17608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 15:09:55.265617 17623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:55.267017 17608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:09:55.267162 17608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "832ab9ba84424d448fc01e9f3e014a63"
format_stamp: "Formatted at 2026-09-25 15:09:55 on dist-test-slave-64fx"
server_key: "e05deb248fe46ac0eeaa7c8c4f101956"
server_key_iv: "14dcf0501a037fbbfe96a16951a3d2c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:55.267375 17608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:55.300364 17608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:55.303828 17608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:55.304111 17608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:55.304944 17608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:55.306209 17608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:55.306277 17608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:55.306437 17608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:55.306486 17608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:55.337702 17608 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34647
I20260925 15:09:55.337723 17735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34647 every 8 connection(s)
I20260925 15:09:55.338832 17608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:55.343628 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17608
I20260925 15:09:55.343793 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:09:55.344228 4919 external_mini_cluster.cc:1428] Setting key ca77c10ea5ce40eac48056a6653a337c
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348995, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:55.357216 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:55.341039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:40501 (local address 127.4.205.254:45099)
0925 15:09:55.341202 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:55.341207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:55.342435 (+ 1228us) server_negotiation.cc:407] Connection header received
0925 15:09:55.344174 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:55.344180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:55.344277 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:55.344392 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:55.346249 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.347312 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.348727 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.349107 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.349198 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:55.352337 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:55.352341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:55.352344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:55.352373 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:55.354898 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.355546 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.355549 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.355551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.355661 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.356014 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.356016 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.356018 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.356249 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:55.356354 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:55.356866 (+ 512us) server_negotiation.cc:299] Negotiation successful
0925 15:09:55.357016 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:09:55.358309 17738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:55.341625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.195:40501)
0925 15:09:55.342288 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:55.342324 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:55.343929 (+ 1605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:55.344657 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:55.344668 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:55.345199 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:55.346051 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:55.346088 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.347440 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.347443 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:55.348613 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:55.348624 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.349269 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.349273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:55.349465 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:55.350131 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:55.350151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:55.352154 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:55.355081 (+ 2927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:55.355087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:55.355097 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:55.355444 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:55.355743 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:55.355746 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:55.355748 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:55.355924 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:55.356340 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:55.356345 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:55.356584 (+ 239us) client_negotiation.cc:769] Sending connection context
0925 15:09:55.356775 (+ 191us) client_negotiation.cc:240] Negotiation successful
0925 15:09:55.357071 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":264,"threads_started":1}
I20260925 15:09:55.359647 17736 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:55.359920 17736 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:55.360543 17736 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:55.361894 17245 ts_manager.cc:194] Registered new tserver with Master: 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647)
I20260925 15:09:55.362900 17245 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:40501
I20260925 15:09:55.371454 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:55.373091 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17326
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:09:55.388156 17740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:55.374055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.4.205.193:34377)
0925 15:09:55.374589 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:55.374596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:55.374618 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:09:55.374669 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:55.374674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:55.374849 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:55.374972 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:55.376246 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.377218 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.378685 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.379178 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.379269 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:55.383054 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:55.383064 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:55.383075 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:55.383118 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:55.385785 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.386410 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.386416 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.386418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.386551 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.386865 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.386868 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.386870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.387165 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:55.387371 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:55.387573 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:09:55.387740 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":282,"threads_started":1}
I20260925 15:09:55.389321 17454 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:55.469836 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17326
I20260925 15:09:55.484937 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17465
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:09:55.500043 17742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:55.485766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51620 (local address 127.4.205.194:44345)
0925 15:09:55.486339 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:55.486346 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:55.486371 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:09:55.486428 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:55.486433 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:55.486654 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:55.486774 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:55.487834 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.488960 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.490520 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.491062 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.491158 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:55.495243 (+ 4085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:55.495253 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:55.495265 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:55.495302 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:55.497853 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.498435 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.498442 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.498444 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.498569 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.498877 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.498880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.498882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.499168 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:55.499359 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:55.499619 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:09:55.499782 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":321,"threads_started":1}
I20260925 15:09:55.501190 17597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:55.587354 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17465
I20260925 15:09:55.602952 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17608
Sep 25 15:09:55 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348992, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:09:55.619190 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:55.603870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58980 (local address 127.4.205.195:34647)
0925 15:09:55.604525 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:55.604532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:55.604559 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:09:55.604615 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:55.604621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:55.604827 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:55.605005 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:55.606341 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.607331 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.608715 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:55.609286 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:55.609436 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:55.613634 (+ 4198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:55.613646 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:55.613660 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:55.613693 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:55.616301 (+ 2608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.617015 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.617022 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.617025 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.617150 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:55.617527 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:55.617538 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:55.617542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:55.617990 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:55.618312 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:55.618585 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:09:55.618842 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":347,"threads_started":1}
I20260925 15:09:55.620414 17731 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:55.698943 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17608
I20260925 15:09:55.713939 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17214
I20260925 15:09:55.715029 17276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:55.815574 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17214
I20260925 15:09:55.837030 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:45099
--webserver_interface=127.4.205.254
--webserver_port=36381
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:45099
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:09:56.134783 17747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:56.135025 17747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:56.135216 17747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:56.142010 17747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:09:56.142122 17747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:56.142207 17747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:56.142303 17747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:09:56.142407 17747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:09:56.151899 17747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:45099
--ranger_config_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:45099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=36381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:56.152974 17747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:56.154202 17747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:56.165100 17752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:56.165124 17753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:56.165319 17755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:56.165925 17747 server_base.cc:1034] running on GCE node
I20260925 15:09:56.166641 17747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:56.167835 17747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:56.169057 17747 hybrid_clock.cc:648] HybridClock initialized: now 1790348996169009 us; error 62 us; skew 500 ppm
I20260925 15:09:56.169370 17747 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:56 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348996, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:56.172230 17747 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:09:56.173485 17747 webserver.cc:460] Webserver started at http://127.4.205.254:36381/ using document root <none> and password file <none>
I20260925 15:09:56.174176 17747 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:56.174315 17747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:09:56 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348996, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:56.241415 17747 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:09:56.245031 17763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:56.246547 17747 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:56.246776 17747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "5027fe8b02f140c6bd0ae6456f3093ef"
format_stamp: "Formatted at 2026-09-25 15:09:52 on dist-test-slave-64fx"
server_key: "849f867809bb37a49367db027e439575"
server_key_iv: "4628a76433640e3d492c6eb365de7a3d"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:56.247705 17747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:56.265970 17747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:56.269434 17747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:56.269747 17747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:56.287307 17814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:45099 every 8 connection(s)
I20260925 15:09:56.287351 17747 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:45099
I20260925 15:09:56.288678 17747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:09:56.294873 17815 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:09:56.297857 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17747
I20260925 15:09:56.297966 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Bootstrap starting.
I20260925 15:09:56.298686 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:34377
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:45099
--webserver_port=34735
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:09:56.315665 17815 log.cc:824] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:56.324072 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:09:56.324702 17815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Bootstrap complete.
I20260925 15:09:56.333411 17815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:56.334297 17815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5027fe8b02f140c6bd0ae6456f3093ef, State: Initialized, Role: FOLLOWER
I20260925 15:09:56.334867 17815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:56.335094 17815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:09:56.335212 17815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:09:56.335405 17815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:09:56.338974 17815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:56.339421 17815 leader_election.cc:302] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5027fe8b02f140c6bd0ae6456f3093ef; no voters:
I20260925 15:09:56.340018 17815 leader_election.cc:288] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:09:56.340128 17819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:09:56.341286 17819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [term 2 LEADER]: Becoming Leader. State: Replica: 5027fe8b02f140c6bd0ae6456f3093ef, State: Running, Role: LEADER
I20260925 15:09:56.341889 17819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } }
I20260925 15:09:56.342162 17815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:09:56.344388 17821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5027fe8b02f140c6bd0ae6456f3093ef. Latest consensus state: current_term: 2 leader_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } } }
I20260925 15:09:56.344786 17821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:56.345386 17820 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5027fe8b02f140c6bd0ae6456f3093ef" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 45099 } } }
I20260925 15:09:56.345762 17820 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef [sys.catalog]: This master's current role is: LEADER
I20260925 15:09:56.350991 17824 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:09:56.356427 17824 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:09:56.357749 17824 catalog_manager.cc:1259] Loaded cluster ID: 4550202c06784cbc89f3e5f9ff3d3888
I20260925 15:09:56.357900 17824 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:09:56.360853 17815 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:09:56.361675 17824 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:09:56.363117 17824 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Loaded TSK: 0
I20260925 15:09:56.364549 17824 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:09:56.727319 17817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:56.727830 17817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:56.728058 17817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:56.738879 17817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:56.739066 17817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:56.739276 17817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:09:56.753968 17817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:34377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=34735
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:56.756876 17817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:56.758602 17817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:56.783854 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:56.785386 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:56.789944 17817 server_base.cc:1034] running on GCE node
W20260925 15:09:56.790621 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:56.791410 17817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:56.796837 17817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:56.798110 17817 hybrid_clock.cc:648] HybridClock initialized: now 1790348996798068 us; error 35 us; skew 500 ppm
I20260925 15:09:56.798522 17817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:56 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348996, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:56.802760 17817 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:09:56.804703 17817 webserver.cc:460] Webserver started at http://127.4.205.193:34735/ using document root <none> and password file <none>
I20260925 15:09:56.805719 17817 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:56.805956 17817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:09:56 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348996, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:56.846755 17817 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:09:56.849809 17863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:56.851696 17817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:56.851927 17817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "66c93d5dd46d4839af68f68af0fc7bc3"
format_stamp: "Formatted at 2026-09-25 15:09:53 on dist-test-slave-64fx"
server_key: "a51bf6578d284386719962372128cbb9"
server_key_iv: "2ba30ff1348de9cd093793867a1be416"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:56.853163 17817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:56.878911 17817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:56.883512 17817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:56.883833 17817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:56.884994 17817 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:56.886656 17817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:56.886772 17817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:56.886904 17817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:56.886984 17817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:57.001742 17817 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:34377
I20260925 15:09:57.001780 17976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:34377 every 8 connection(s)
I20260925 15:09:57.003330 17817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:09:57.014154 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17817
I20260925 15:09:57.014932 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:44345
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:45099
--webserver_port=42341
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:57 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348996, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:57.042141 17980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:57.008059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.193:47161)
0925 15:09:57.010679 (+ 2620us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:57.010730 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:57.013047 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:57.025958 (+ 12911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:57.025970 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:57.026493 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:57.027055 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:57.027072 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.028631 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.028642 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:57.030085 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:57.030107 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.031262 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.031271 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:57.031537 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:57.032199 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:57.032220 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:57.034189 (+ 1969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:57.037441 (+ 3252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:57.037452 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:57.037467 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:57.037961 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:57.038546 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:57.038553 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:57.038558 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:57.038758 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:57.039470 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:57.039478 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:57.039790 (+ 312us) client_negotiation.cc:769] Sending connection context
0925 15:09:57.040034 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 15:09:57.040659 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2326,"thread_start_us":9467,"threads_started":1}
I20260925 15:09:57.042140 17979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:57.007884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:47161 (local address 127.4.205.254:45099)
0925 15:09:57.024584 (+ 16700us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:57.024598 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:57.024639 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:09:57.025000 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:57.025014 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:57.025383 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:57.025817 (+ 434us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:57.027280 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.028463 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.030239 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.031027 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.031231 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:57.034435 (+ 3204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:57.034443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:57.034456 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:57.034490 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:57.037297 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:57.038167 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:57.038173 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:57.038178 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:57.038335 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:57.038879 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:57.038887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:57.038902 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:57.039329 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:57.039523 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:57.040309 (+ 786us) server_negotiation.cc:299] Negotiation successful
0925 15:09:57.040667 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16460,"thread_start_us":16270,"threads_started":1}
I20260925 15:09:57.045543 17977 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:57.045840 17977 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:57.046428 17977 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
W20260925 15:09:57.049015 17977 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:45099 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:09:57.049233 17977 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:57.049475 17977 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:57.050591 17977 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:57.050889 17977 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
W20260925 15:09:57.051975 17977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:09:57.404824 17981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:57.405121 17981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:57.405346 17981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:57.412456 17981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:57.412596 17981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:57.412770 17981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:09:57.422555 17981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:44345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=42341
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:57.423791 17981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:57.425251 17981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:57.438941 17994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:57.439200 17993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:57.439353 17996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:57.439951 17981 server_base.cc:1034] running on GCE node
I20260925 15:09:57.440448 17981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:57.441463 17981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:57.442755 17981 hybrid_clock.cc:648] HybridClock initialized: now 1790348997442678 us; error 47 us; skew 500 ppm
I20260925 15:09:57.443063 17981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:57 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:57.446148 17981 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:09:57.453378 17981 webserver.cc:460] Webserver started at http://127.4.205.194:42341/ using document root <none> and password file <none>
I20260925 15:09:57.454229 17981 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:57.454380 17981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:09:57 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:57.488695 17981 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260925 15:09:57.492306 18006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:57.494035 17981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:09:57.494208 17981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "3fcaad890ec04ee19ba2bcd2e9560f73"
format_stamp: "Formatted at 2026-09-25 15:09:54 on dist-test-slave-64fx"
server_key: "67e5b70f1323b9d4cc289ea86da5c572"
server_key_iv: "ee052aa58802cc289783db0d61e2f8aa"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:57.495198 17981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:57.514263 17981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:57.519225 17981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:57.519538 17981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:57.532050 17981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:57.543149 17981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:57.543286 17981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:57.543459 17981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:57.543553 17981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:57.613255 17981 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:44345
I20260925 15:09:57.613300 18120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:44345 every 8 connection(s)
I20260925 15:09:57.614751 17981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:09:57.615907 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 17981
I20260925 15:09:57.616519 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.4.205.212:50835/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:34647
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:45099
--webserver_port=43067
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:33927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:09:57 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
Sep 25 15:09:57 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:57.650926 18123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:57.617888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:50969 (local address 127.4.205.254:45099)
0925 15:09:57.629903 (+ 12015us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:57.629911 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:57.633225 (+ 3314us) server_negotiation.cc:407] Connection header received
0925 15:09:57.633296 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:57.633301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:57.633384 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:57.633485 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:57.635474 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.636467 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.638920 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.639750 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.639865 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:57.644120 (+ 4255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:57.644128 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:57.644134 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:57.644179 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:57.647575 (+ 3396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:57.648335 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:57.648340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:57.648344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:57.648475 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:57.648983 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:57.648989 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:57.648995 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:57.649343 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:57.649468 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:57.650339 (+ 871us) server_negotiation.cc:299] Negotiation successful
0925 15:09:57.650534 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11872,"thread_start_us":11758,"threads_started":1}
I20260925 15:09:57.653355 18124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:57.618936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.194:50969)
0925 15:09:57.630426 (+ 11490us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:57.630480 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:57.632885 (+ 2405us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:57.633927 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:57.633939 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:57.634494 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:57.635289 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:57.635315 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.636664 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.636669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:57.638713 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:57.638729 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:57.640181 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:57.640188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:57.640427 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:57.641231 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:57.641285 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:57.643897 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:57.647710 (+ 3813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:57.647717 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:57.647732 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:57.648196 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:57.648605 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:57.648609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:57.648612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:57.648849 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:57.649547 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:57.649579 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:57.649952 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 15:09:57.651482 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0925 15:09:57.652017 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11212,"thread_start_us":11012,"threads_started":1}
I20260925 15:09:57.655161 18121 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:57.655437 18121 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:57.656196 18121 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:57.658116 17780 ts_manager.cc:194] Registered new tserver with Master: 3fcaad890ec04ee19ba2bcd2e9560f73 (127.4.205.194:44345)
I20260925 15:09:57.660130 17780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:50969
W20260925 15:09:58.041102 18125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:09:58.041399 18125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:09:58.041607 18125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:09:58.050470 18125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:09:58.050583 18125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:09:58.050776 18125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:09:58.052116 17977 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:58.052520 17977 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:58.053738 17780 ts_manager.cc:194] Registered new tserver with Master: 66c93d5dd46d4839af68f68af0fc7bc3 (127.4.205.193:34377)
I20260925 15:09:58.054893 17780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:47161
I20260925 15:09:58.061704 18125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:33927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.4.205.212:50835/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:34647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=43067
--tserver_master_addrs=127.4.205.254:45099
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:09:58.063114 18125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:09:58.064208 18125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:09:58.080643 18131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:58.080623 18132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:09:58.088735 18134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:09:58.089071 18125 server_base.cc:1034] running on GCE node
I20260925 15:09:58.089753 18125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:09:58.090704 18125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:09:58.091989 18125 hybrid_clock.cc:648] HybridClock initialized: now 1790348998091945 us; error 58 us; skew 500 ppm
I20260925 15:09:58.092306 18125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:09:58 dist-test-slave-64fx krb5kdc[16770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348998, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:09:58.095384 18125 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:09:58.098284 18125 webserver.cc:460] Webserver started at http://127.4.205.195:43067/ using document root <none> and password file <none>
I20260925 15:09:58.099018 18125 fs_manager.cc:359] Metadata directory not provided
I20260925 15:09:58.099143 18125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:09:58 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348998, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
Sep 25 15:09:58 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:58.165840 18125 fs_manager.cc:709] Time spent opening directory manager: real 0.012s user 0.010s sys 0.000s
I20260925 15:09:58.180862 18142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:09:58.182915 18125 fs_manager.cc:725] Time spent opening block manager: real 0.015s user 0.010s sys 0.004s
I20260925 15:09:58.183187 18125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "832ab9ba84424d448fc01e9f3e014a63"
format_stamp: "Formatted at 2026-09-25 15:09:55 on dist-test-slave-64fx"
server_key: "e05deb248fe46ac0eeaa7c8c4f101956"
server_key_iv: "14dcf0501a037fbbfe96a16951a3d2c9"
server_key_version: "kuduclusterkey@0"
I20260925 15:09:58.184638 18125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:09:58.215605 18125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:09:58.220234 18125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:09:58.220595 18125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:09:58.233377 18125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:09:58.241393 18125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:09:58.241613 18125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 15:09:58.241779 18125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:09:58.241884 18125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 15:09:58 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348997, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for HTTP/127.4.205.212@KRBTEST.COM
I20260925 15:09:58.664429 18121 heartbeater.cc:503] Master 127.4.205.254:45099 was elected leader, sending a full tablet report...
I20260925 15:09:58.859041 18125 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34647
I20260925 15:09:58.859099 18262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34647 every 8 connection(s)
I20260925 15:09:58.860939 18125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:09:58.863276 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18125
I20260925 15:09:58.916553 18265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:58.865604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.4.205.254:45099)
0925 15:09:58.868829 (+ 3225us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:58.868839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:58.883894 (+ 15055us) server_negotiation.cc:407] Connection header received
0925 15:09:58.883968 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:58.883973 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:58.884061 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:58.884171 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:58.886319 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.887400 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.889886 (+ 2486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.890488 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.890600 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:58.897353 (+ 6753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:58.897358 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:58.897363 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:58.897397 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:58.912386 (+ 14989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.914326 (+ 1940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.914333 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.914338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.914501 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.914942 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.914946 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.914949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.915343 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:58.915611 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:58.915963 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:09:58.916165 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3075,"thread_start_us":2961,"threads_started":1}
Sep 25 15:09:58 dist-test-slave-64fx krb5kdc[16770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348998, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:09:58.940279 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:58.877235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:37769 (local address 127.4.205.254:45099)
0925 15:09:58.903144 (+ 25909us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:58.903150 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:58.906544 (+ 3394us) server_negotiation.cc:407] Connection header received
0925 15:09:58.906593 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:58.906597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:58.906673 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:58.906762 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:58.914896 (+ 8134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.916116 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.919429 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.920049 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.920164 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:58.924623 (+ 4459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:58.924630 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:58.924634 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:58.924669 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:58.936615 (+ 11946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.937522 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.937527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.937530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.937715 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.938202 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.938207 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.938210 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.938502 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:58.938629 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:58.939708 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0925 15:09:58.939939 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25788,"thread_start_us":25913,"threads_started":1}
I20260925 15:09:58.941752 18267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:58.876694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:45099 (local address 127.4.205.195:37769)
0925 15:09:58.903755 (+ 27061us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:58.903827 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:58.906139 (+ 2312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:58.907929 (+ 1790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:58.907940 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:58.908542 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:09:58.909383 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:58.909406 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.916298 (+ 6892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.916304 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:58.919078 (+ 2774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:58.919094 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.920253 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.920260 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:58.920519 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:58.921235 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:09:58.921276 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:09:58.924245 (+ 2969us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:09:58.936813 (+ 12568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:58.936821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:58.936840 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:58.937394 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:58.937845 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:09:58.937849 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:09:58.937853 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:09:58.938084 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:09:58.938698 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:09:58.938708 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:09:58.939240 (+ 532us) client_negotiation.cc:769] Sending connection context
0925 15:09:58.939516 (+ 276us) client_negotiation.cc:240] Negotiation successful
0925 15:09:58.940023 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26739,"thread_start_us":30056,"threads_started":1}
I20260925 15:09:58.943177 18263 heartbeater.cc:347] Connected to a master server at 127.4.205.254:45099
I20260925 15:09:58.943503 18263 heartbeater.cc:464] Registering TS with master...
I20260925 15:09:58.944406 18263 heartbeater.cc:511] Master 127.4.205.254:45099 requested a full tablet report, sending...
I20260925 15:09:58.946081 17780 ts_manager.cc:194] Registered new tserver with Master: 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647)
I20260925 15:09:58.947193 17780 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:37769
I20260925 15:09:58.950984 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:09:58.970719 18268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:58.954708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.4.205.254:45099)
0925 15:09:58.954869 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:58.954875 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:58.954993 (+ 118us) server_negotiation.cc:407] Connection header received
0925 15:09:58.955241 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:58.955247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:58.955330 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:58.955426 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:58.957343 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.958499 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.960461 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:58.961069 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:58.961194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:58.964364 (+ 3170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:58.964370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:58.964374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:58.964408 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:58.967825 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.968637 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.968642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.968646 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.968783 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:58.969237 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:58.969242 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:58.969246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:58.969603 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:58.969830 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:58.970178 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 15:09:58.970387 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:09:58.976356 17780 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:09:58.987368 17780 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:09:59.058121 17977 heartbeater.cc:503] Master 127.4.205.254:45099 was elected leader, sending a full tablet report...
I20260925 15:09:59.067698 18278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.008738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34647 (local address 127.0.0.1:58986)
0925 15:09:59.044844 (+ 36106us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.044860 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.058409 (+ 13549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.059558 (+ 1149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.059565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.059593 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.059902 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.059931 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.062628 (+ 2697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.062633 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.064940 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.064954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.066710 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.066716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.066915 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.066965 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.067104 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.067295 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35970,"thread_start_us":34780,"threads_started":1}
I20260925 15:09:59.068475 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.009333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58986 (local address 127.4.205.195:34647)
0925 15:09:59.041682 (+ 32349us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.041692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.044957 (+ 3265us) server_negotiation.cc:407] Connection header received
0925 15:09:59.058722 (+ 13765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.058730 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.058994 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.059146 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.060768 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.062430 (+ 1662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.065107 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.066497 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.066641 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.066910 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.067978 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.068146 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32102,"thread_start_us":32377,"threads_started":1}
I20260925 15:09:59.070173 18280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.006802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39174)
0925 15:09:59.043076 (+ 36274us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.043122 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.043310 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.062970 (+ 19660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.062976 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.063003 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.063379 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.063391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.065326 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.065330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.067409 (+ 2079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.067428 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.069342 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.069348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.069509 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.069550 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.069649 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.069787 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35970,"mutex_wait_us":55,"thread_start_us":38180,"threads_started":1}
I20260925 15:09:59.070956 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.007048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39174 (local address 127.4.205.193:34377)
0925 15:09:59.062380 (+ 55332us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.062388 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.062412 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:09:59.062514 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.062523 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.062750 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.063600 (+ 850us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.063806 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.065191 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.067583 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.069139 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.069290 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.069498 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.070464 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.070618 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55182,"thread_start_us":55047,"threads_started":1}
I20260925 15:09:59.072470 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.051562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44345 (local address 127.0.0.1:51622)
0925 15:09:59.057917 (+ 6355us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.057934 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.058044 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.062371 (+ 4327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.062376 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.062402 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.062824 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.062839 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.065896 (+ 3057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.065901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.069852 (+ 3951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.069865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.071706 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.071712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.071892 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.071927 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.072016 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.072169 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6250,"thread_start_us":6553,"threads_started":1}
I20260925 15:09:59.073110 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.045660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.4.205.194:44345)
0925 15:09:59.060911 (+ 15251us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.060924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.060980 (+ 56us) server_negotiation.cc:407] Connection header received
0925 15:09:59.061095 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.061106 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.061436 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.061695 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.063025 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.064611 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.070027 (+ 5416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.071495 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.071657 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.071875 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.072666 (+ 791us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.072842 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15022,"thread_start_us":14825,"threads_started":1}
I20260925 15:09:59.072387 18191 tablet_service.cc:1467] Processing CreateTablet for tablet 740f7e73a9fb43e881b55268febd1b5d (DEFAULT_TABLE table=test-table [id=31faa6135df74815aa820d9c479f814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:59.075381 18191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740f7e73a9fb43e881b55268febd1b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:59.075268 17911 tablet_service.cc:1467] Processing CreateTablet for tablet 740f7e73a9fb43e881b55268febd1b5d (DEFAULT_TABLE table=test-table [id=31faa6135df74815aa820d9c479f814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:59.076514 17911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740f7e73a9fb43e881b55268febd1b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:59.076540 18056 tablet_service.cc:1467] Processing CreateTablet for tablet 740f7e73a9fb43e881b55268febd1b5d (DEFAULT_TABLE table=test-table [id=31faa6135df74815aa820d9c479f814b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:09:59.077773 18056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 740f7e73a9fb43e881b55268febd1b5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:09:59.098549 18284 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: Bootstrap starting.
I20260925 15:09:59.100378 18286 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Bootstrap starting.
I20260925 15:09:59.100677 18285 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Bootstrap starting.
I20260925 15:09:59.103768 18284 tablet_bootstrap.cc:654] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:59.104317 18285 tablet_bootstrap.cc:654] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:59.104986 18284 log.cc:824] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:59.105578 18285 log.cc:824] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:59.107434 18286 tablet_bootstrap.cc:654] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Neither blocks nor log segments found. Creating new log.
I20260925 15:09:59.108830 18286 log.cc:824] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Log is configured to *not* fsync() on all Append() calls
I20260925 15:09:59.114991 18285 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: No bootstrap required, opened a new log
I20260925 15:09:59.115000 18286 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: No bootstrap required, opened a new log
I20260925 15:09:59.115046 18284 tablet_bootstrap.cc:492] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: No bootstrap required, opened a new log
I20260925 15:09:59.115298 18286 ts_tablet_manager.cc:1397] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260925 15:09:59.115298 18285 ts_tablet_manager.cc:1397] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.006s
I20260925 15:09:59.115453 18284 ts_tablet_manager.cc:1397] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.012s
I20260925 15:09:59.125531 18284 raft_consensus.cc:348] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.125536 18285 raft_consensus.cc:348] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.125950 18285 raft_consensus.cc:374] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:59.126078 18285 raft_consensus.cc:729] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fcaad890ec04ee19ba2bcd2e9560f73, State: Initialized, Role: FOLLOWER
I20260925 15:09:59.126084 18284 raft_consensus.cc:374] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:59.126221 18284 raft_consensus.cc:729] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66c93d5dd46d4839af68f68af0fc7bc3, State: Initialized, Role: FOLLOWER
I20260925 15:09:59.126746 18285 consensus_queue.cc:260] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.126945 18284 consensus_queue.cc:260] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.127633 18286 raft_consensus.cc:348] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.128175 18286 raft_consensus.cc:374] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:09:59.128320 18286 raft_consensus.cc:729] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 832ab9ba84424d448fc01e9f3e014a63, State: Initialized, Role: FOLLOWER
I20260925 15:09:59.129091 18286 consensus_queue.cc:260] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.134881 18284 ts_tablet_manager.cc:1428] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: Time spent starting tablet: real 0.019s user 0.009s sys 0.008s
I20260925 15:09:59.142001 18290 raft_consensus.cc:482] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:59.142302 18290 raft_consensus.cc:504] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.143292 18285 ts_tablet_manager.cc:1428] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Time spent starting tablet: real 0.028s user 0.017s sys 0.010s
I20260925 15:09:59.145135 18286 ts_tablet_manager.cc:1428] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Time spent starting tablet: real 0.030s user 0.018s sys 0.004s
I20260925 15:09:59.145605 18263 heartbeater.cc:503] Master 127.4.205.254:45099 was elected leader, sending a full tablet report...
I20260925 15:09:59.145669 18290 leader_election.cc:288] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66c93d5dd46d4839af68f68af0fc7bc3 (127.4.205.193:34377), 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647)
I20260925 15:09:59.154241 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.145579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:51321 (local address 127.4.205.195:34647)
0925 15:09:59.145800 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.145806 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.146819 (+ 1013us) server_negotiation.cc:407] Connection header received
0925 15:09:59.147112 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.147120 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.147204 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.147292 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.148772 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.149851 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.152204 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.153314 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.153460 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.153524 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.153875 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.153995 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 15:09:59.154304 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.144794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34647 (local address 127.4.205.194:51321)
0925 15:09:59.146709 (+ 1915us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.146732 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.146949 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.147552 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.147559 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.147588 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.148083 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.148097 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.149968 (+ 1871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.149976 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.151992 (+ 2016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.152007 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.153502 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.153508 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.153700 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.153735 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.153841 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.153964 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1768,"thread_start_us":3563,"threads_started":1}
I20260925 15:09:59.155256 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.145243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:57637 (local address 127.4.205.193:34377)
0925 15:09:59.145439 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.145445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.148675 (+ 3230us) server_negotiation.cc:407] Connection header received
0925 15:09:59.148853 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.148859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.148922 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.148977 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.149653 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.150778 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.152989 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.154082 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.154216 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.154286 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.154906 (+ 620us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.155025 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:09:59.155308 18294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.144823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.4.205.194:57637)
0925 15:09:59.148590 (+ 3767us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.148606 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.148716 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.149143 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.149147 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.149169 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.149497 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.149507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.150906 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.150912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.152861 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.152871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.154460 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.154467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.154636 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.154669 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.154883 (+ 214us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.155001 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3665,"thread_start_us":1641,"threads_started":1}
I20260925 15:09:59.155354 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832ab9ba84424d448fc01e9f3e014a63" is_pre_election: true
I20260925 15:09:59.155809 18211 raft_consensus.cc:2454] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fcaad890ec04ee19ba2bcd2e9560f73 in term 0.
I20260925 15:09:59.156042 18291 raft_consensus.cc:482] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:09:59.156181 17931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" is_pre_election: true
I20260925 15:09:59.156291 18291 raft_consensus.cc:504] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.156850 18010 leader_election.cc:302] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fcaad890ec04ee19ba2bcd2e9560f73, 832ab9ba84424d448fc01e9f3e014a63; no voters:
I20260925 15:09:59.156951 17931 raft_consensus.cc:2454] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3fcaad890ec04ee19ba2bcd2e9560f73 in term 0.
I20260925 15:09:59.157243 18290 raft_consensus.cc:2781] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:09:59.157362 18290 raft_consensus.cc:482] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:09:59.157441 18290 raft_consensus.cc:3037] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:59.158036 18291 leader_election.cc:288] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3fcaad890ec04ee19ba2bcd2e9560f73 (127.4.205.194:44345), 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647)
I20260925 15:09:59.160475 18290 raft_consensus.cc:504] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.161121 18290 leader_election.cc:288] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [CANDIDATE]: Term 1 election: Requested vote from peers 66c93d5dd46d4839af68f68af0fc7bc3 (127.4.205.193:34377), 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647)
I20260925 15:09:59.161652 17931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66c93d5dd46d4839af68f68af0fc7bc3"
I20260925 15:09:59.161912 17931 raft_consensus.cc:3037] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:09:59.162601 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832ab9ba84424d448fc01e9f3e014a63"
I20260925 15:09:59.162763 18211 raft_consensus.cc:3037] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:09:59.165107 18266 tablet.cc:2369] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:09:59.165817 18211 raft_consensus.cc:2454] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fcaad890ec04ee19ba2bcd2e9560f73 in term 1.
I20260925 15:09:59.165975 17931 raft_consensus.cc:2454] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3fcaad890ec04ee19ba2bcd2e9560f73 in term 1.
I20260925 15:09:59.166172 18010 leader_election.cc:302] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3fcaad890ec04ee19ba2bcd2e9560f73, 832ab9ba84424d448fc01e9f3e014a63; no voters:
I20260925 15:09:59.166409 18290 raft_consensus.cc:2781] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:09:59.166591 18290 raft_consensus.cc:686] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 1 LEADER]: Becoming Leader. State: Replica: 3fcaad890ec04ee19ba2bcd2e9560f73, State: Running, Role: LEADER
I20260925 15:09:59.167063 18290 consensus_queue.cc:237] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } }
I20260925 15:09:59.172528 17780 catalog_manager.cc:5495] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3fcaad890ec04ee19ba2bcd2e9560f73 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 } health_report { overall_health: UNKNOWN } } }
I20260925 15:09:59.176543 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.159514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:44385 (local address 127.4.205.195:34647)
0925 15:09:59.165683 (+ 6169us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.165688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.168939 (+ 3251us) server_negotiation.cc:407] Connection header received
0925 15:09:59.169336 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.169341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.169409 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.169474 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.170229 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.171224 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.173781 (+ 2557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.174656 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.174749 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.174795 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.176154 (+ 1359us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.176276 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6067}
I20260925 15:09:59.177095 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.161709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50257 (local address 127.4.205.194:44345)
0925 15:09:59.161987 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.161993 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.162009 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:59.162057 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.162061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.162120 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.162199 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:09:59.171515 (+ 9316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.172567 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.174923 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.176062 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.176199 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.176259 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.176746 (+ 487us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.176874 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163}
I20260925 15:09:59.177482 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.159156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34647 (local address 127.4.205.193:44385)
0925 15:09:59.168852 (+ 9696us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.168866 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.168997 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.169693 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.169697 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.169722 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.170043 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.170059 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.171427 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.171432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.173600 (+ 2168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.173614 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.175804 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.175811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.175972 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.176006 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.176813 (+ 807us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.176928 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9596,"mutex_wait_us":27,"thread_start_us":9907,"threads_started":1}
I20260925 15:09:59.177887 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.158540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44345 (local address 127.4.205.193:50257)
0925 15:09:59.159141 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.159163 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:09:59.159276 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:09:59.169042 (+ 9766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:09:59.169050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:09:59.169085 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:09:59.169454 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.169467 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.172739 (+ 3272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.172744 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.174770 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:09:59.174784 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.176390 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.176396 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:09:59.176551 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.176594 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:09:59.177280 (+ 686us) client_negotiation.cc:240] Negotiation successful
0925 15:09:59.177422 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"mutex_wait_us":3,"thread_start_us":1319,"threads_started":1}
I20260925 15:09:59.178150 18211 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "832ab9ba84424d448fc01e9f3e014a63" is_pre_election: true
I20260925 15:09:59.178422 18211 raft_consensus.cc:2379] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 66c93d5dd46d4839af68f68af0fc7bc3 in current term 1: Already voted for candidate 3fcaad890ec04ee19ba2bcd2e9560f73 in this term.
I20260925 15:09:59.178561 18076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "740f7e73a9fb43e881b55268febd1b5d" candidate_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3fcaad890ec04ee19ba2bcd2e9560f73" is_pre_election: true
I20260925 15:09:59.179487 17864 leader_election.cc:302] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 66c93d5dd46d4839af68f68af0fc7bc3; no voters: 3fcaad890ec04ee19ba2bcd2e9560f73, 832ab9ba84424d448fc01e9f3e014a63
I20260925 15:09:59.179863 18291 raft_consensus.cc:2726] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:09:59.210548 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.204576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.4.205.194:44345)
0925 15:09:59.204823 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.204831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.205064 (+ 233us) server_negotiation.cc:407] Connection header received
0925 15:09:59.205260 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.205264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.205344 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.205456 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:09:59.205863 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.206532 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.208407 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.208772 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.208879 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.209425 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:09:59.209976 (+ 551us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:09:59.210077 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.210204 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.210261 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:09:59.217935 18211 raft_consensus.cc:1261] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 1 FOLLOWER]: Refusing update from remote peer 3fcaad890ec04ee19ba2bcd2e9560f73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:59.217978 17931 raft_consensus.cc:1261] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 1 FOLLOWER]: Refusing update from remote peer 3fcaad890ec04ee19ba2bcd2e9560f73: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:09:59.218750 18293 consensus_queue.cc:1035] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "832ab9ba84424d448fc01e9f3e014a63" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:59.219089 18290 consensus_queue.cc:1035] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66c93d5dd46d4839af68f68af0fc7bc3" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:09:59.244650 18302 mvcc.cc:205] Tried to move back new op lower bound from 7333269500787503104 to 7333269500592373760. Current Snapshot: MvccSnapshot[applied={T|T < 7333269500787503104}]
I20260925 15:09:59.251322 17780 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:59.251569 17780 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59868:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:09:59.255709 17780 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5027fe8b02f140c6bd0ae6456f3093ef: Sending DeleteTablet for 3 replicas of tablet 740f7e73a9fb43e881b55268febd1b5d
I20260925 15:09:59.256724 17911 tablet_service.cc:1514] Processing DeleteTablet for tablet 740f7e73a9fb43e881b55268febd1b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39174
I20260925 15:09:59.260641 18191 tablet_service.cc:1514] Processing DeleteTablet for tablet 740f7e73a9fb43e881b55268febd1b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58986
I20260925 15:09:59.261818 18056 tablet_service.cc:1514] Processing DeleteTablet for tablet 740f7e73a9fb43e881b55268febd1b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:51622
I20260925 15:09:59.269205 18190 tablet_service.cc:1514] Processing DeleteTablet for tablet 740f7e73a9fb43e881b55268febd1b5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:09:59 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:58986
I20260925 15:09:59.274621 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17817
I20260925 15:09:59.297093 18311 tablet_replica.cc:330] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: stopping tablet replica
I20260925 15:09:59.297583 18311 raft_consensus.cc:2229] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:09:59.298370 18311 raft_consensus.cc:2258] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:59.305768 18309 tablet_replica.cc:330] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3: stopping tablet replica
I20260925 15:09:59.306310 18309 raft_consensus.cc:2229] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:59.306906 18309 raft_consensus.cc:2258] T 740f7e73a9fb43e881b55268febd1b5d P 66c93d5dd46d4839af68f68af0fc7bc3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:59.307915 18279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.275686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39182 (local address 127.4.205.193:34377)
0925 15:09:59.275866 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.275872 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.275956 (+ 84us) server_negotiation.cc:407] Connection header received
0925 15:09:59.276142 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.276147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.276235 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.276335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:59.283688 (+ 7353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.284442 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.286661 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.287228 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.287340 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.300745 (+ 13405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:59.300772 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:59.300785 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:59.300816 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:59.304391 (+ 3575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.305195 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.305205 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.305208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.305352 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.305784 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.305788 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.305792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.306336 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:59.306687 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.307299 (+ 612us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.307537 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:09:59.309520 17972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:59.309582 18311 ts_tablet_manager.cc:1905] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:59.314216 18310 tablet_replica.cc:330] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: stopping tablet replica
I20260925 15:09:59.314545 18310 raft_consensus.cc:2229] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:09:59.314929 18310 raft_consensus.cc:2258] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:09:59.319171 18310 ts_tablet_manager.cc:1905] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:09:59.326038 18310 ts_tablet_manager.cc:1918] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:59.326226 18310 log.cc:1196] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/740f7e73a9fb43e881b55268febd1b5d
I20260925 15:09:59.326787 18310 ts_tablet_manager.cc:1939] T 740f7e73a9fb43e881b55268febd1b5d P 832ab9ba84424d448fc01e9f3e014a63: Deleting consensus metadata
I20260925 15:09:59.327869 18311 ts_tablet_manager.cc:1918] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:09:59.328078 18311 log.cc:1196] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/740f7e73a9fb43e881b55268febd1b5d
I20260925 15:09:59.328634 18311 ts_tablet_manager.cc:1939] T 740f7e73a9fb43e881b55268febd1b5d P 3fcaad890ec04ee19ba2bcd2e9560f73: Deleting consensus metadata
I20260925 15:09:59.328809 17767 catalog_manager.cc:4841] TS 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647): tablet 740f7e73a9fb43e881b55268febd1b5d (table test-table [id=31faa6135df74815aa820d9c479f814b]) successfully deleted
W20260925 15:09:59.329879 17767 catalog_manager.cc:4808] TS 832ab9ba84424d448fc01e9f3e014a63 (127.4.205.195:34647): delete failed for tablet 740f7e73a9fb43e881b55268febd1b5d because the tablet was not found. No further retry: Not found: Tablet not found: 740f7e73a9fb43e881b55268febd1b5d
I20260925 15:09:59.330329 17764 catalog_manager.cc:4841] TS 3fcaad890ec04ee19ba2bcd2e9560f73 (127.4.205.194:44345): tablet 740f7e73a9fb43e881b55268febd1b5d (table test-table [id=31faa6135df74815aa820d9c479f814b]) successfully deleted
I20260925 15:09:59.414618 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17817
I20260925 15:09:59.430962 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17981
W20260925 15:09:59.431030 17764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.205.193:34377 (error 108)
I20260925 15:09:59.432958 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.432481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39194)
0925 15:09:59.432660 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.432773 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260925 15:09:59.433189 17764 catalog_manager.cc:4584] TS 66c93d5dd46d4839af68f68af0fc7bc3 (127.4.205.193:34377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 740f7e73a9fb43e881b55268febd1b5d: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
I20260925 15:09:59.449247 18282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.432119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51640 (local address 127.4.205.194:44345)
0925 15:09:59.432363 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.432371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.432388 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:09:59.432451 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.432456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.432533 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.432632 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:59.434167 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.434896 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.436139 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.436545 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.436666 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.443658 (+ 6992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:59.443677 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:59.443690 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:59.443725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:59.446588 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.447176 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.447193 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.447196 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.447327 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.447744 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.447750 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.447758 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.448150 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:59.448491 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.448726 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.448948 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:09:59.450135 18116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:59.486665 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.486183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39204)
0925 15:09:59.486368 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.486473 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:09:59.487630 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.487264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39210)
0925 15:09:59.487415 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.487494 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:09:59.547978 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.547460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39214)
0925 15:09:59.547678 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.547789 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 15:09:59.548916 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.548545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39218)
0925 15:09:59.548690 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.548777 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:09:59.560318 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17981
I20260925 15:09:59.577440 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18125
I20260925 15:09:59.589109 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.578100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58996 (local address 127.4.205.195:34647)
0925 15:09:59.578248 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:09:59.578253 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:09:59.578269 (+ 16us) server_negotiation.cc:407] Connection header received
0925 15:09:59.578387 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:09:59.578391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:09:59.578464 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:09:59.578542 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:09:59.579734 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.580380 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.581674 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:09:59.582055 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:09:59.582144 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:09:59.584516 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:09:59.584529 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:09:59.584542 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:09:59.584570 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:09:59.587084 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.587652 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.587659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.587661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.587765 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:09:59.588042 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:09:59.588045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:09:59.588047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:09:59.588364 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:09:59.588581 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0925 15:09:59.588743 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:09:59.588879 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:09:59.589921 18258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:59.629969 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.629490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39228)
0925 15:09:59.629717 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.629810 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:09:59.630924 18283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:09:59.630551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34377 (local address 127.0.0.1:39244)
0925 15:09:59.630737 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:09:59.630800 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:34377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260925 15:09:59.675557 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18125
I20260925 15:09:59.691243 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 17747
I20260925 15:09:59.692296 17810 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:09:59.802652 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 17747
I20260925 15:09:59.827376 4919 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:10:00.162045 4919 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:10:00.162261 4919 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:10:00.508230 4919 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:10:00.508 UTC [16800] LOG: received smart shutdown request
2026-09-25 15:10:00.511 UTC [16800] LOG: background worker "logical replication launcher" (PID 16809) exited with exit code 1
2026-09-25 15:10:00.512 UTC [16804] LOG: shutting down
2026-09-25 15:10:00.658 UTC [16800] LOG: database system is shut down
2026-09-25T15:10:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (50968 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:10:00 dist-test-slave-64fx krb5kdc[18325](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:10:00 dist-test-slave-64fx krb5kdc[18325](info): set up 2 sockets
Sep 25 15:10:00 dist-test-slave-64fx krb5kdc[18325](info): commencing operation
krb5kdc: starting...
W20260925 15:10:01.829485 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:10:01 dist-test-slave-64fx krb5kdc[18325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:10:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:10:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:10:02.026609 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40171
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:02.326516 18338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:02.326833 18338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:02.326984 18338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:02.334084 18338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:02.334205 18338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:02.334300 18338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:02.334386 18338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:02.334460 18338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:02.343365 18338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:02.344563 18338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:02.345772 18338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:02.356209 18343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:02.356166 18346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:02.356232 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:02.357349 18338 server_base.cc:1034] running on GCE node
I20260925 15:10:02.358044 18338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:02.359191 18338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:02.360376 18338 hybrid_clock.cc:648] HybridClock initialized: now 1790349002360338 us; error 24 us; skew 500 ppm
I20260925 15:10:02.360718 18338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:02 dist-test-slave-64fx krb5kdc[18325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:02.363569 18338 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:02.364841 18338 webserver.cc:460] Webserver started at http://127.4.205.254:44867/ using document root <none> and password file <none>
I20260925 15:10:02.365461 18338 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:02.365608 18338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:02.365883 18338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:02.368494 18338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "8089d3cd26cd44ad9568dffd32b19bb8"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "67f7f713329eb60017534eebb897ce9c"
server_key_iv: "7440af4d07fac96f429e6465ad2f65f7"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.369210 18338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "8089d3cd26cd44ad9568dffd32b19bb8"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "67f7f713329eb60017534eebb897ce9c"
server_key_iv: "7440af4d07fac96f429e6465ad2f65f7"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.373869 18338 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.004s
I20260925 15:10:02.376793 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:02.377969 18338 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:02.378130 18338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "8089d3cd26cd44ad9568dffd32b19bb8"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "67f7f713329eb60017534eebb897ce9c"
server_key_iv: "7440af4d07fac96f429e6465ad2f65f7"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.378304 18338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:02.400398 18338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:02.406728 18338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:02.407264 18338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:02.427784 18338 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:40171
I20260925 15:10:02.427817 18404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:40171 every 8 connection(s)
I20260925 15:10:02.429003 18338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:02.432571 18405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:02.435372 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18338
I20260925 15:10:02.435582 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:10:02.436281 4919 external_mini_cluster.cc:1428] Setting key 4ddddd3918b49c2a3d7964c192bde4b6
I20260925 15:10:02.440935 18405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:10:02 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.253@KRBTEST.COM" created.
I20260925 15:10:02.462976 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.444929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.4.205.254:40171)
0925 15:10:02.445791 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:02.445804 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:02.446022 (+ 218us) server_negotiation.cc:407] Connection header received
0925 15:10:02.447896 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:02.447902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:02.448110 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:02.448299 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:02.449500 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.450625 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.451953 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.452498 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.452678 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.456158 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:02.456164 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:02.456173 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:02.456201 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:02.458787 (+ 2586us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.459470 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.459481 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.459488 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.459661 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.460037 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.460042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.460048 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.460479 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:02.460676 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:02.461138 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 15:10:02.461560 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":630,"thread_start_us":372,"threads_started":1}
I20260925 15:10:02.462992 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.444929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40171 (local address 127.0.0.1:42236)
0925 15:10:02.445885 (+ 956us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.445916 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:02.447540 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:02.448314 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:02.448320 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:02.448959 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:02.449321 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.449339 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.450764 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.450771 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.451815 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.451830 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.452711 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.452718 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.452902 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.453649 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:02.453673 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:02.455963 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:02.458917 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.458929 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.458941 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.459306 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.459783 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.459789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.459793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.459940 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.460603 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:02.460615 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:02.460801 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 15:10:02.461032 (+ 231us) client_negotiation.cc:240] Negotiation successful
0925 15:10:02.461691 (+ 659us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"mutex_wait_us":161,"spinlock_wait_cycles":40320,"thread_start_us":471,"threads_started":1}
I20260925 15:10:02.465833 18405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:02.466738 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.466413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59590)
0925 15:10:02.466554 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.466652 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:10:02.466930 18356 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
I20260925 15:10:02.468286 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.467983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59606)
0925 15:10:02.468115 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.468181 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:10:02.468485 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:38569: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
I20260925 15:10:02.484688 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:38569
--webserver_interface=127.4.205.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:02.517810 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } attempt: 1
I20260925 15:10:02.518889 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.518444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59616)
0925 15:10:02.518636 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.518765 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:10:02.519877 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.519547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59622)
0925 15:10:02.519671 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.519733 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:10:02.520121 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:38569: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
I20260925 15:10:02.588428 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } attempt: 2
I20260925 15:10:02.589476 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.589089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59634)
0925 15:10:02.589253 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.589358 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 15:10:02.590390 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.590089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59644)
0925 15:10:02.590222 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.590279 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:10:02.590574 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:38569: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
I20260925 15:10:02.681785 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } attempt: 3
I20260925 15:10:02.682927 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.682526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59654)
0925 15:10:02.682701 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.682790 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:10:02.683881 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.683584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59660)
0925 15:10:02.683692 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.683760 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:10:02.684130 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:38569: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
W20260925 15:10:02.789490 18412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:02.789783 18412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:02.789934 18412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:02.796547 18412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:02.796627 18412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:02.796667 18412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:02.796730 18412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:02.796787 18412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:02.805337 18412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:38569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.205.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:02.806476 18412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:02.807544 18412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:02.817065 18417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:02.818079 18420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:02.818075 18418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:02.819156 18412 server_base.cc:1034] running on GCE node
I20260925 15:10:02.819741 18412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:02.820806 18412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:02.821981 18412 hybrid_clock.cc:648] HybridClock initialized: now 1790349002821946 us; error 29 us; skew 500 ppm
I20260925 15:10:02.822250 18412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:02 dist-test-slave-64fx krb5kdc[18325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:02.824849 18412 init.cc:377] Logged in from keytab as kudu/127.4.205.253@KRBTEST.COM (short username kudu)
I20260925 15:10:02.826048 18412 webserver.cc:460] Webserver started at http://127.4.205.253:44681/ using document root <none> and password file <none>
I20260925 15:10:02.826630 18412 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:02.826754 18412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:02.826999 18412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:02.827355 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } attempt: 4
I20260925 15:10:02.828383 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.827877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59668)
0925 15:10:02.828142 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.828237 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260925 15:10:02.829278 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.828943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59672)
0925 15:10:02.829074 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.829156 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:10:02.829447 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:38569: Network error: Client connection negotiation failed: client connection to 127.4.205.253:38569: connect: Connection refused (error 111)
I20260925 15:10:02.830157 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data/instance:
uuid: "575a9b9f01b3470bbc9425eca10dafbb"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "718011c25fcf2432beb2b6fd7246b85d"
server_key_iv: "7cae57ff6e1103322d098ad7ac9c815b"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.830790 18412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal/instance:
uuid: "575a9b9f01b3470bbc9425eca10dafbb"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "718011c25fcf2432beb2b6fd7246b85d"
server_key_iv: "7cae57ff6e1103322d098ad7ac9c815b"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.835088 18412 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:10:02.837994 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:02.839085 18412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:10:02.839200 18412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "575a9b9f01b3470bbc9425eca10dafbb"
format_stamp: "Formatted at 2026-09-25 15:10:02 on dist-test-slave-64fx"
server_key: "718011c25fcf2432beb2b6fd7246b85d"
server_key_iv: "7cae57ff6e1103322d098ad7ac9c815b"
server_key_version: "encryptionkey@0"
I20260925 15:10:02.839383 18412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:02.859536 18412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:02.863190 18412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:02.863461 18412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:02.879967 18412 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.253:38569
I20260925 15:10:02.880028 18478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.253:38569 every 8 connection(s)
I20260925 15:10:02.881174 18412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
I20260925 15:10:02.882668 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18412
I20260925 15:10:02.882862 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-1/wal/instance
I20260925 15:10:02.883380 4919 external_mini_cluster.cc:1428] Setting key 5baa3be875e50e1894989cd7586c9277
I20260925 15:10:02.885506 18479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:02.893540 18479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.4.205.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.252@KRBTEST.COM" created.
Sep 25 15:10:02 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:02.913656 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.895906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42244 (local address 127.4.205.254:40171)
0925 15:10:02.896069 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:02.896073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:02.897219 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 15:10:02.899044 (+ 1825us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:02.899050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:02.899132 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:02.899237 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:02.900661 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.901360 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.902868 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.903251 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.903368 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.907752 (+ 4384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:02.907756 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:02.907759 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:02.907788 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:02.910381 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.911307 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.911311 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.911313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.911416 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.911864 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.911867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.911869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.912116 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:02.912223 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:02.912955 (+ 732us) server_negotiation.cc:299] Negotiation successful
0925 15:10:02.913206 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:10:02.914285 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.896406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40171 (local address 127.0.0.1:42244)
0925 15:10:02.897075 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.897109 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:02.898789 (+ 1680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:02.899458 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:02.899466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:02.899915 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:02.900503 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.900519 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.901525 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.901528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.902729 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.902740 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.903457 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.903463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.903688 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.904293 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:02.904313 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:02.907555 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:02.910532 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.910543 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.910561 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.911171 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.911513 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.911521 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.911527 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.911754 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.912242 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:02.912253 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:02.912583 (+ 330us) client_negotiation.cc:769] Sending connection context
0925 15:10:02.912794 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 15:10:02.913133 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":289,"threads_started":1}
I20260925 15:10:02.915977 18479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:10:02 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:10:02.931015 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.916730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59674)
0925 15:10:02.916878 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.916896 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:02.916993 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:02.918181 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:02.918184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:02.918496 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:02.918639 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.918646 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.920601 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.920604 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.921411 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:02.921421 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.922676 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.922680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:02.922882 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.923429 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:02.923452 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:02.925239 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:02.928658 (+ 3419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.928662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.928665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.929089 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.929463 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:02.929467 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:02.929469 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:02.929660 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:02.930258 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:02.930261 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:02.930406 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:10:02.930521 (+ 115us) client_negotiation.cc:240] Negotiation successful
0925 15:10:02.930690 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":34}
I20260925 15:10:02.931131 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.917014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.4.205.253:38569)
0925 15:10:02.917716 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:02.917726 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:02.917755 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:10:02.917806 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:02.917816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:02.918032 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:02.918182 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:02.918792 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.920483 (+ 1691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.921545 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:02.922529 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:02.922667 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:02.925463 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:02.925475 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:02.925491 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:02.925526 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:02.928493 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.929209 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.929215 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.929216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.929319 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:02.929742 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:02.929745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:02.929747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:02.930148 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:02.930348 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:02.930618 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 15:10:02.930793 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":502,"thread_start_us":342,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
I20260925 15:10:02.933487 18479 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:02.934476 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:42471
--webserver_interface=127.4.205.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:02.934579 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.934106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58038)
0925 15:10:02.934282 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.934407 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:10:02.934865 18429 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:02.936331 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.935960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58048)
0925 15:10:02.936109 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.936202 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260925 15:10:02.936515 18479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:02.985785 18479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 1
I20260925 15:10:02.986711 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.986349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58064)
0925 15:10:02.986507 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.986591 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:10:02.987579 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:02.987306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58066)
0925 15:10:02.987438 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:02.987495 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:10:02.987802 18479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:03.056098 18479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 2
I20260925 15:10:03.057139 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.056739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58080)
0925 15:10:03.056905 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.057000 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:10:03.058156 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.057878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58086)
0925 15:10:03.058002 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.058060 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:10:03.058416 18479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:03.128798 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } attempt: 5
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:10:03.141392 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.129827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59676 (local address 127.4.205.253:38569)
0925 15:10:03.130101 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.130106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.130121 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:10:03.130187 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.130190 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.130248 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.130357 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.131136 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.131812 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.132970 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.133424 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.133554 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.136535 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.136539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.136542 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.136573 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.139079 (+ 2506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.139646 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.139653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.139656 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.139780 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.140113 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.140120 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.140124 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.140366 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.140501 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.140881 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.141078 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260925 15:10:03.141389 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.129533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59676)
0925 15:10:03.129724 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.129744 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.129960 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.130398 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.130404 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.130795 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.130979 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.130988 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.131945 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.131952 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.132821 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.132833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.133616 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.133622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.133798 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.134413 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.134431 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.136353 (+ 1922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.139213 (+ 2860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.139218 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.139223 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.139534 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.139878 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.139884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.139888 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.140018 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.140488 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.140492 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.140637 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.140782 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.141023 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:10:03.142576 18405 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:03.143517 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.143124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58092)
0925 15:10:03.143263 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.143347 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:10:03.144326 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.144040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58094)
0925 15:10:03.144162 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.144241 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:10:03.144510 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:03.149614 18479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 3
I20260925 15:10:03.150323 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.149995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58098)
0925 15:10:03.150130 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.150205 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 15:10:03.151206 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.150850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58108)
0925 15:10:03.151043 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.151111 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260925 15:10:03.151389 18479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:03.195799 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 1
I20260925 15:10:03.196794 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.196377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58118)
0925 15:10:03.196564 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.196650 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:10:03.197747 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.197431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58122)
0925 15:10:03.197586 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.197660 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 15:10:03.197944 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
W20260925 15:10:03.233599 18486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:03.233911 18486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:03.234050 18486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:03.240849 18486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:03.240969 18486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:03.241061 18486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:03.241149 18486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:03.241230 18486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:03.250408 18486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:40171,127.4.205.253:38569,127.4.205.252:42471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:42471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.205.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:03.251670 18486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:03.252879 18486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:03.263592 18494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:03.263664 18491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:03.263659 18492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:03.264410 18486 server_base.cc:1034] running on GCE node
I20260925 15:10:03.265071 18486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:03.266126 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 2
I20260925 15:10:03.266387 18486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:03.267161 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.266704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58138)
0925 15:10:03.266928 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.267015 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:10:03.267598 18486 hybrid_clock.cc:648] HybridClock initialized: now 1790349003267547 us; error 49 us; skew 500 ppm
I20260925 15:10:03.267963 18486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:10:03.268136 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.267751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58148)
0925 15:10:03.267907 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.267997 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:10:03.268364 18405 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349003, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:03.270898 18486 init.cc:377] Logged in from keytab as kudu/127.4.205.252@KRBTEST.COM (short username kudu)
I20260925 15:10:03.272202 18486 webserver.cc:460] Webserver started at http://127.4.205.252:36359/ using document root <none> and password file <none>
I20260925 15:10:03.272841 18486 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:03.272991 18486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:03.273277 18486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:03.276051 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data/instance:
uuid: "d74acfe6bb0f4ac6997e2e77dee5c102"
format_stamp: "Formatted at 2026-09-25 15:10:03 on dist-test-slave-64fx"
server_key: "49a21a45d23349acc8bd90ddaeab50cf"
server_key_iv: "91c2d1077e9914e7aeee0c6cdf9ac213"
server_key_version: "encryptionkey@0"
I20260925 15:10:03.276793 18486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal/instance:
uuid: "d74acfe6bb0f4ac6997e2e77dee5c102"
format_stamp: "Formatted at 2026-09-25 15:10:03 on dist-test-slave-64fx"
server_key: "49a21a45d23349acc8bd90ddaeab50cf"
server_key_iv: "91c2d1077e9914e7aeee0c6cdf9ac213"
server_key_version: "encryptionkey@0"
I20260925 15:10:03.281780 18486 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 15:10:03.284865 18501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:03.286077 18486 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:03.286229 18486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "d74acfe6bb0f4ac6997e2e77dee5c102"
format_stamp: "Formatted at 2026-09-25 15:10:03 on dist-test-slave-64fx"
server_key: "49a21a45d23349acc8bd90ddaeab50cf"
server_key_iv: "91c2d1077e9914e7aeee0c6cdf9ac213"
server_key_version: "encryptionkey@0"
I20260925 15:10:03.286425 18486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:03.294575 18479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 4
I20260925 15:10:03.295702 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.295106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58152)
0925 15:10:03.295321 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.295523 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:10:03.296612 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.296284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58164)
0925 15:10:03.296411 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.296488 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260925 15:10:03.296813 18479 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:42471: Network error: Client connection negotiation failed: client connection to 127.4.205.252:42471: connect: Connection refused (error 111)
I20260925 15:10:03.321179 18486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:03.324795 18486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:03.325047 18486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:03.341905 18486 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.252:42471
I20260925 15:10:03.341920 18552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.252:42471 every 8 connection(s)
I20260925 15:10:03.343250 18486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
I20260925 15:10:03.346542 18553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:03.352876 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18486
I20260925 15:10:03.353044 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348808911781-4919-0/minicluster-data/master-2/wal/instance
I20260925 15:10:03.352253 18553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:03.353545 4919 external_mini_cluster.cc:1428] Setting key 6388306ff8196386e297baf784817ae5
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349003, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:10:03.370970 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.355384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42258 (local address 127.4.205.254:40171)
0925 15:10:03.355613 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.355638 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.356264 (+ 626us) server_negotiation.cc:407] Connection header received
0925 15:10:03.357870 (+ 1606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.357875 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.357947 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.358046 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.359419 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.360054 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.361456 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.361928 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.362054 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.365276 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.365284 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.365291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.365336 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.368067 (+ 2731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.368784 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.368791 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.368795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.368919 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.369294 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.369299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.369303 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.369657 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.369816 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.370517 (+ 701us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.370723 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:10:03.372145 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.355194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:40171 (local address 127.0.0.1:42258)
0925 15:10:03.356115 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.356165 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.357641 (+ 1476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.358459 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.358463 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.358930 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.359268 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.359282 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.360202 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.360207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.361297 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.361326 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.362141 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.362151 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.362330 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.362990 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.363008 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.365096 (+ 2088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.368198 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.368210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.368224 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.368667 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.369017 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.369022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.369026 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.369185 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.369777 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.369782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.370109 (+ 327us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.370414 (+ 305us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.370841 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":453,"threads_started":1}
I20260925 15:10:03.372319 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.355193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58170 (local address 127.4.205.252:42471)
0925 15:10:03.356161 (+ 968us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.356170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.356202 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:10:03.357636 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.357654 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.357898 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.358078 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.359295 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.360350 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.361729 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.362229 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.362369 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.366717 (+ 4348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.366723 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.366732 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.366759 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.369310 (+ 2551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.370082 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.370089 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.370092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.370230 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.370652 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.370658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.370662 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.371058 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.371326 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.371687 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.371906 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":576,"threads_started":1}
I20260925 15:10:03.373685 18553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:03.374826 18405 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 3
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349003, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:03.388013 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.374383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59688 (local address 127.4.205.253:38569)
0925 15:10:03.374567 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.374573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.374777 (+ 204us) server_negotiation.cc:407] Connection header received
0925 15:10:03.374940 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.374949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.375024 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.375117 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.376010 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.376784 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.378459 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.378858 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.378984 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.382468 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.382477 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.382497 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.382554 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.385316 (+ 2762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.386052 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.386060 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.386064 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.386193 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.386596 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.386601 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.386606 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.386849 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.387012 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.387416 (+ 404us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.387634 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:10:03.388819 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.374444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:38569 (local address 127.0.0.1:59688)
0925 15:10:03.374688 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.374704 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.374814 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.375276 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.375282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.375679 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.375860 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.375874 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.376932 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.376939 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.378346 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.378355 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.378983 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.378988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.379164 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.379646 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.379665 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.382195 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.385463 (+ 3268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.385469 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.385474 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.385936 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.386313 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.386319 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.386323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.386473 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.386966 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.386970 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.387126 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.388248 (+ 1122us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.388506 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 15:10:03.389762 18553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:10:03.389854 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.375376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58176)
0925 15:10:03.375613 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.375629 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.375697 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.376387 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.376392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.376771 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.376988 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.377000 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.378295 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.378300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.379662 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.379677 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.380580 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.380585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.380766 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.381372 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.381388 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.384349 (+ 2961us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.387569 (+ 3220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.387575 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.387580 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.387935 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.388317 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.388322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.388326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.388436 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.389021 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.389027 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.389202 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.389324 (+ 122us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.389479 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133}
I20260925 15:10:03.390622 18557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.375991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58176 (local address 127.4.205.252:42471)
0925 15:10:03.376130 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.376135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.376148 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:10:03.376185 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.376189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.376242 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.376328 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.377107 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.378164 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.379801 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.380414 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.380608 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.384522 (+ 3914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.384528 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.384534 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.384589 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.387449 (+ 2860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.388046 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.388052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.388056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.388198 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.388533 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.388538 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.388542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.388892 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.390043 (+ 1151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.390158 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.390360 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:10:03.391448 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.375112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.4.205.254:40171)
0925 15:10:03.375260 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.375265 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.375388 (+ 123us) server_negotiation.cc:407] Connection header received
0925 15:10:03.375537 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.375541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.375605 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.375693 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.377020 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.377914 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.379430 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.379991 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.380129 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.384658 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.384665 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.384672 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.384707 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.388565 (+ 3858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.389445 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.389450 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.389453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.389658 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.390080 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.390086 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.390091 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.390399 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.390644 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.390922 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.391122 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:10:03.395808 18405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: Bootstrap starting.
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349003, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:10:03.399036 18405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:03.400050 18405 log.cc:824] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:03.402472 18556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.390302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58192)
0925 15:10:03.391012 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.391028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.391118 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.391412 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.391418 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.391909 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.392036 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.392043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.393102 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.393109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.394000 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.394025 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.394692 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.394697 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.394903 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.395382 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.395395 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.397257 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.399794 (+ 2537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.399799 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.399802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.400267 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.400591 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.400595 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.400599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.400749 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.401457 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.401461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.401651 (+ 190us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.401799 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.402066 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"mutex_wait_us":20}
I20260925 15:10:03.402958 18405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: No bootstrap required, opened a new log
I20260925 15:10:03.403311 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.390534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58192 (local address 127.4.205.252:42471)
0925 15:10:03.391103 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.391108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.391131 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:10:03.391207 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.391212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.391268 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.391390 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.392195 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.392999 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.394185 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.394571 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.394685 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.397426 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.397430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.397433 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.397456 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.399675 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.400364 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.400368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.400370 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.400469 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.400893 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.400901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.400907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.401239 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.401410 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.402779 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.402965 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":451,"thread_start_us":313,"threads_started":1}
I20260925 15:10:03.407573 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: Bootstrap starting.
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:10:03.410427 18405 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.410816 18405 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:03.410934 18405 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8089d3cd26cd44ad9568dffd32b19bb8, State: Initialized, Role: FOLLOWER
I20260925 15:10:03.411012 18553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:03.411520 18405 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.412186 18553 log.cc:824] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:03.412736 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.413153 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.414006 18405 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:03.416389 18553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: No bootstrap required, opened a new log
I20260925 15:10:03.418992 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.401120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59704 (local address 127.4.205.253:38569)
0925 15:10:03.401290 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.401298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.401419 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:10:03.401614 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.401619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.401681 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.401779 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.403264 (+ 1485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.404316 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.405896 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.406481 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.406592 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.411479 (+ 4887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.411485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.411489 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.411524 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.415434 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.416506 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.416514 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.416520 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.416706 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.417167 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.417174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.417180 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.417650 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.417987 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.418219 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.418507 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:10:03.424238 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:10:03.425962 18553 raft_consensus.cc:348] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.426342 18553 raft_consensus.cc:374] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:03.426448 18553 raft_consensus.cc:729] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74acfe6bb0f4ac6997e2e77dee5c102, State: Initialized, Role: FOLLOWER
I20260925 15:10:03.427163 18553 consensus_queue.cc:260] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.434074 18574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
W20260925 15:10:03.435844 18572 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 15:10:03.435833 18553 sys_catalog.cc:564] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 15:10:03.435961 18572 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:10:03.437114 18574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.430214 18338 master_runner.cc:186] Error getting master registration for 127.4.205.253:38569: OK, instance_id { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" instance_seqno: 1790349002875384 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:10:03.441962 18338 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:10:03.479125 18408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.452128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42276 (local address 127.4.205.254:40171)
0925 15:10:03.452310 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.452317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.455508 (+ 3191us) server_negotiation.cc:407] Connection header received
0925 15:10:03.456368 (+ 860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.456373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.456444 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.456535 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.461332 (+ 4797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.462507 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.466912 (+ 4405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.467582 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.467676 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.472257 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.472263 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.472270 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.472310 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.475822 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.476654 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.476660 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.476663 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.476820 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.477275 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.477280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.477283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.477675 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.477931 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.478514 (+ 583us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.478763 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:10:03.492355 18484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.452479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.4.205.253:38569)
0925 15:10:03.452668 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.452675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.465875 (+ 13200us) server_negotiation.cc:407] Connection header received
0925 15:10:03.466574 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.466579 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.466650 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.466716 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.469104 (+ 2388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.470130 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.473899 (+ 3769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.476120 (+ 2221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.476245 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.484466 (+ 8221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.484474 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.484477 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.484521 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.488249 (+ 3728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.489089 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.489095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.489098 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.489240 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.490325 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.490330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.490333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.490722 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.490957 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.491777 (+ 820us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.492004 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260925 15:10:03.494933 18592 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:10:03.495103 18592 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:10:03.498165 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.455679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58194 (local address 127.4.205.252:42471)
0925 15:10:03.455888 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.455896 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.466393 (+ 10497us) server_negotiation.cc:407] Connection header received
0925 15:10:03.466604 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.466609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.466678 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.466741 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.468914 (+ 2173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.470027 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.477391 (+ 7364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.478342 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.478459 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.490319 (+ 11860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.490325 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.490330 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.490365 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.494926 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.495689 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.495695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.495699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.495856 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.496315 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.496320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.496325 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.496720 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.496975 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.497524 (+ 549us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.497788 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:10:03.487916 18486 master_runner.cc:186] Error getting master registration for 127.4.205.253:38569: OK, instance_id { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" instance_seqno: 1790349002875384 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:10:03.501938 18486 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:10:03.576820 18507 tablet.cc:2369] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:10:03.596002 18479 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } attempt: 5
Sep 25 15:10:03 dist-test-slave-64fx krb5kdc[18325](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349002, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:10:03.608979 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.596537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:42471 (local address 127.0.0.1:58208)
0925 15:10:03.596761 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:03.596780 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:03.596925 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:03.597242 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:03.597247 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:03.597732 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:03.597885 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.597893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.598892 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.598900 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.599743 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:03.599753 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.600442 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.600449 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:03.600665 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.601264 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:03.601285 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:03.603353 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:03.606488 (+ 3135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.606493 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.606499 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.606849 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.607221 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:03.607226 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:03.607249 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:03.607397 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:03.608011 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:03.608015 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:03.608183 (+ 168us) client_negotiation.cc:769] Sending connection context
0925 15:10:03.608296 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:10:03.608569 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260925 15:10:03.610080 18558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:03.596580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58208 (local address 127.4.205.252:42471)
0925 15:10:03.596741 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:03.596746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:03.596855 (+ 109us) server_negotiation.cc:407] Connection header received
0925 15:10:03.597046 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:03.597060 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:03.597120 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:03.597220 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:03.598033 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.598767 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.599877 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:03.600294 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:03.600426 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:03.603542 (+ 3116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:03.603546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:03.603550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:03.603579 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:03.606354 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.606969 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.606978 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.606981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.607107 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:03.607546 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:03.607551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:03.607553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:03.607884 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:03.609296 (+ 1412us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:03.609438 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:10:03.609722 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:10:03.614789 18479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: Bootstrap starting.
I20260925 15:10:03.617990 18479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:03.618980 18479 log.cc:824] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:03.621438 18479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: No bootstrap required, opened a new log
I20260925 15:10:03.630395 18479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.630692 18479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:03.630821 18479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 575a9b9f01b3470bbc9425eca10dafbb, State: Initialized, Role: FOLLOWER
I20260925 15:10:03.631402 18479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.632280 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.632709 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.633423 18479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:03.654223 18412 master_runner.cc:263] No leader master found from master 575a9b9f01b3470bbc9425eca10dafbb
I20260925 15:10:03.654364 18412 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 15:10:03.656761 18606 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:10:03.656931 18606 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260925 15:10:03.660553 18359 tablet.cc:2369] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:10:03.687517 18560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:03.687892 18560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.690395 18454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a9b9f01b3470bbc9425eca10dafbb" is_pre_election: true
I20260925 15:10:03.691018 18454 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8089d3cd26cd44ad9568dffd32b19bb8 in term 0.
I20260925 15:10:03.691684 18356 leader_election.cc:302] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 575a9b9f01b3470bbc9425eca10dafbb, 8089d3cd26cd44ad9568dffd32b19bb8; no voters:
I20260925 15:10:03.692895 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" is_pre_election: true
I20260925 15:10:03.693490 18528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8089d3cd26cd44ad9568dffd32b19bb8 in term 0.
I20260925 15:10:03.689512 18560 leader_election.cc:288] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 575a9b9f01b3470bbc9425eca10dafbb (127.4.205.253:38569), d74acfe6bb0f4ac6997e2e77dee5c102 (127.4.205.252:42471)
I20260925 15:10:03.695815 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:10:03.695971 18560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:10:03.696081 18560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:03.699739 18560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.700805 18528 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102"
I20260925 15:10:03.701030 18528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:03.704351 18528 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8089d3cd26cd44ad9568dffd32b19bb8 in term 1.
I20260925 15:10:03.704758 18355 leader_election.cc:302] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8089d3cd26cd44ad9568dffd32b19bb8, d74acfe6bb0f4ac6997e2e77dee5c102; no voters:
I20260925 15:10:03.707638 18560 leader_election.cc:288] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [CANDIDATE]: Term 1 election: Requested vote from peers 575a9b9f01b3470bbc9425eca10dafbb (127.4.205.253:38569), d74acfe6bb0f4ac6997e2e77dee5c102 (127.4.205.252:42471)
I20260925 15:10:03.707849 18560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:03.708004 18560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [term 1 LEADER]: Becoming Leader. State: Replica: 8089d3cd26cd44ad9568dffd32b19bb8, State: Running, Role: LEADER
I20260925 15:10:03.708536 18560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } }
I20260925 15:10:03.734517 18454 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "575a9b9f01b3470bbc9425eca10dafbb"
I20260925 15:10:03.734751 18454 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:03.738322 18454 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8089d3cd26cd44ad9568dffd32b19bb8 in term 1.
I20260925 15:10:03.740326 18608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8089d3cd26cd44ad9568dffd32b19bb8. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.740715 18608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:03.748560 18610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:03.750980 18610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:03.769670 18528 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [term 1 FOLLOWER]: Refusing update from remote peer 8089d3cd26cd44ad9568dffd32b19bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:03.770699 18608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:03.777750 18454 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [term 1 FOLLOWER]: Refusing update from remote peer 8089d3cd26cd44ad9568dffd32b19bb8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:03.778467 18608 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:03.811187 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8089d3cd26cd44ad9568dffd32b19bb8. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.811546 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.815359 18611 mvcc.cc:205] Tried to move back new op lower bound from 7333269519430729728 to 7333269519201996800. Current Snapshot: MvccSnapshot[applied={T|T < 7333269519430729728}]
I20260925 15:10:03.819918 18613 mvcc.cc:205] Tried to move back new op lower bound from 7333269519430729728 to 7333269519201996800. Current Snapshot: MvccSnapshot[applied={T|T < 7333269519430729728}]
I20260925 15:10:03.820500 18560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.820797 18560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:03.823423 18610 catalog_manager.cc:1347] Generated new cluster ID: 15bcbe6dbeb246d396899a08db873bae
I20260925 15:10:03.823544 18610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:03.825630 18574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8089d3cd26cd44ad9568dffd32b19bb8. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.825871 18574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.829015 18608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.829198 18608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:03.830667 18612 mvcc.cc:205] Tried to move back new op lower bound from 7333269519430729728 to 7333269519201996800. Current Snapshot: MvccSnapshot[applied={T|T < 7333269519430729728}]
I20260925 15:10:03.835103 18595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.835343 18595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.839856 18574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8089d3cd26cd44ad9568dffd32b19bb8" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40171 } } peers { permanent_uuid: "575a9b9f01b3470bbc9425eca10dafbb" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 38569 } } peers { permanent_uuid: "d74acfe6bb0f4ac6997e2e77dee5c102" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 42471 } } }
I20260925 15:10:03.840073 18574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:10:03.867780 18610 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:10:03.869679 18610 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:03.890394 18610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8089d3cd26cd44ad9568dffd32b19bb8: Generated new TSK 0
I20260925 15:10:03.891296 18610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:10:04.497056 18592 catalog_manager.cc:1259] Loaded cluster ID: 15bcbe6dbeb246d396899a08db873bae
I20260925 15:10:04.497149 18592 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: loading cluster ID for follower catalog manager: success
I20260925 15:10:04.499509 18592 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: acquiring CA information for follower catalog manager: success
I20260925 15:10:04.501180 18592 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d74acfe6bb0f4ac6997e2e77dee5c102: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:10:04.658877 18606 catalog_manager.cc:1259] Loaded cluster ID: 15bcbe6dbeb246d396899a08db873bae
I20260925 15:10:04.659008 18606 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: loading cluster ID for follower catalog manager: success
I20260925 15:10:04.661367 18606 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: acquiring CA information for follower catalog manager: success
I20260925 15:10:04.663163 18606 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 575a9b9f01b3470bbc9425eca10dafbb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:10:05.031837 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18338
I20260925 15:10:05.033048 18400 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:05.142159 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18338
I20260925 15:10:05.161976 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18412
I20260925 15:10:05.162982 18474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:05.263732 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18412
I20260925 15:10:05.283015 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18486
I20260925 15:10:05.284082 18548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:05.388064 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18486
2026-09-25T15:10:05Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4613 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:10:05 dist-test-slave-64fx krb5kdc[18629](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:10:05 dist-test-slave-64fx krb5kdc[18629](info): set up 2 sockets
Sep 25 15:10:05 dist-test-slave-64fx krb5kdc[18629](info): commencing operation
krb5kdc: starting...
W20260925 15:10:06.454177 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:10:06 dist-test-slave-64fx krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:10:06.678375 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41455
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:07.001425 18642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:07.001755 18642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:07.001868 18642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:07.008944 18642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:07.009039 18642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:07.009092 18642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:07.009155 18642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:07.009207 18642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:07.018720 18642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:07.019955 18642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:07.021163 18642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:07.033010 18650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:07.033054 18648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:07.033077 18647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:07.033915 18642 server_base.cc:1034] running on GCE node
I20260925 15:10:07.034581 18642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:07.035825 18642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:07.037016 18642 hybrid_clock.cc:648] HybridClock initialized: now 1790349007036974 us; error 37 us; skew 500 ppm
I20260925 15:10:07.037349 18642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:07 dist-test-slave-64fx krb5kdc[18629](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349007, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:07.040688 18642 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:07.042105 18642 webserver.cc:460] Webserver started at http://127.4.205.254:38433/ using document root <none> and password file <none>
I20260925 15:10:07.042718 18642 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:07.042819 18642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:07.043064 18642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:07.045822 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d28a721e4f394d7e9eab89630f25147e"
format_stamp: "Formatted at 2026-09-25 15:10:07 on dist-test-slave-64fx"
server_key: "5b879e86d50b741a595176e0e22cfb4b"
server_key_iv: "c648bbccb105dfd0d68dccd06a692dca"
server_key_version: "encryptionkey@0"
I20260925 15:10:07.046536 18642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d28a721e4f394d7e9eab89630f25147e"
format_stamp: "Formatted at 2026-09-25 15:10:07 on dist-test-slave-64fx"
server_key: "5b879e86d50b741a595176e0e22cfb4b"
server_key_iv: "c648bbccb105dfd0d68dccd06a692dca"
server_key_version: "encryptionkey@0"
I20260925 15:10:07.051281 18642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:10:07.054348 18657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:07.055552 18642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:07.055722 18642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d28a721e4f394d7e9eab89630f25147e"
format_stamp: "Formatted at 2026-09-25 15:10:07 on dist-test-slave-64fx"
server_key: "5b879e86d50b741a595176e0e22cfb4b"
server_key_iv: "c648bbccb105dfd0d68dccd06a692dca"
server_key_version: "encryptionkey@0"
I20260925 15:10:07.055889 18642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:07.084906 18642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:07.088675 18642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:07.088946 18642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:07.106606 18642 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41455
I20260925 15:10:07.106630 18708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41455 every 8 connection(s)
I20260925 15:10:07.107849 18642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:07.109194 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18642
I20260925 15:10:07.109412 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:10:07.110217 4919 external_mini_cluster.cc:1428] Setting key 71adb4acff215e30737b5ccac806d161
I20260925 15:10:07.112128 18709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:10:07 dist-test-slave-64fx krb5kdc[18629](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:07.125113 18709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e: Bootstrap starting.
I20260925 15:10:07.128206 18709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:07.129191 18709 log.cc:824] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:07.131750 18709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e: No bootstrap required, opened a new log
I20260925 15:10:07.132720 18712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:07.111889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53616 (local address 127.4.205.254:41455)
0925 15:10:07.112682 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:07.112693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:07.112731 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:10:07.114357 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:07.114377 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:07.114736 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:07.115055 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:07.116802 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:07.118218 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:07.120131 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:07.120741 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:07.120898 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:07.125810 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:07.125817 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:07.125829 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:07.125867 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:07.128793 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:07.129614 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:07.129621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:07.129631 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:07.129798 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:07.130215 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:07.130219 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:07.130221 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:07.130677 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:07.130947 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:07.131402 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 15:10:07.131823 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":603,"thread_start_us":376,"threads_started":1}
I20260925 15:10:07.136466 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:10:07.140049 18709 raft_consensus.cc:348] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } }
I20260925 15:10:07.140338 18709 raft_consensus.cc:374] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:07.140444 18709 raft_consensus.cc:729] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d28a721e4f394d7e9eab89630f25147e, State: Initialized, Role: FOLLOWER
I20260925 15:10:07.140926 18709 consensus_queue.cc:260] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } }
I20260925 15:10:07.141129 18709 raft_consensus.cc:388] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:07.141281 18709 raft_consensus.cc:482] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:07.141475 18709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:07.143452 18709 raft_consensus.cc:504] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } }
I20260925 15:10:07.143903 18709 leader_election.cc:302] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d28a721e4f394d7e9eab89630f25147e; no voters:
I20260925 15:10:07.144410 18709 leader_election.cc:288] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:10:07.144519 18714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:07.145145 18714 raft_consensus.cc:686] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [term 1 LEADER]: Becoming Leader. State: Replica: d28a721e4f394d7e9eab89630f25147e, State: Running, Role: LEADER
I20260925 15:10:07.145668 18714 consensus_queue.cc:237] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } }
I20260925 15:10:07.146272 18709 sys_catalog.cc:564] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:07.148128 18715 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d28a721e4f394d7e9eab89630f25147e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } } }
I20260925 15:10:07.148541 18715 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:07.148118 18716 sys_catalog.cc:455] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d28a721e4f394d7e9eab89630f25147e. Latest consensus state: current_term: 1 leader_uuid: "d28a721e4f394d7e9eab89630f25147e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d28a721e4f394d7e9eab89630f25147e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41455 } } }
I20260925 15:10:07.148933 18716 sys_catalog.cc:458] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:07.150429 18720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:07.155545 18720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:07.163304 18720 catalog_manager.cc:1347] Generated new cluster ID: ea74d6f842214cc2adfb6233a21b36ce
I20260925 15:10:07.163408 18720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:07.169349 18720 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:10:07.170441 18720 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:07.175266 18720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d28a721e4f394d7e9eab89630f25147e: Generated new TSK 0
I20260925 15:10:07.175873 18720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:10:07.195631 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18642
I20260925 15:10:07.196580 18704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:07.283833 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18642
2026-09-25T15:10:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1887 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:10:07 dist-test-slave-64fx krb5kdc[18737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:10:07 dist-test-slave-64fx krb5kdc[18737](info): set up 2 sockets
Sep 25 15:10:07 dist-test-slave-64fx krb5kdc[18737](info): commencing operation
krb5kdc: starting...
W20260925 15:10:08.321926 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:10:08 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:10:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:10:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:10:08.528693 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:08.837991 18750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:08.838239 18750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:08.838424 18750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:08.845448 18750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:08.845534 18750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:08.845700 18750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:08.845811 18750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:08.845896 18750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:08.855059 18750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:08.856223 18750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:08.857450 18750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:08.868659 18755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:08.868692 18758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:08.868753 18756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:08.869815 18750 server_base.cc:1034] running on GCE node
I20260925 15:10:08.870435 18750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:08.871626 18750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:08.872819 18750 hybrid_clock.cc:648] HybridClock initialized: now 1790349008872763 us; error 48 us; skew 500 ppm
I20260925 15:10:08.873101 18750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:08 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:08.875994 18750 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:08.877384 18750 webserver.cc:460] Webserver started at http://127.4.205.254:39781/ using document root <none> and password file <none>
I20260925 15:10:08.878073 18750 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:08.878201 18750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:08.878494 18750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:08.881237 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "30d2734b82834aa798d17851baf983ea"
format_stamp: "Formatted at 2026-09-25 15:10:08 on dist-test-slave-64fx"
server_key: "a8f7ef904d41d70953a150c6456cffb6"
server_key_iv: "75fed6adac2b96f47aae9c8735290240"
server_key_version: "encryptionkey@0"
I20260925 15:10:08.882064 18750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "30d2734b82834aa798d17851baf983ea"
format_stamp: "Formatted at 2026-09-25 15:10:08 on dist-test-slave-64fx"
server_key: "a8f7ef904d41d70953a150c6456cffb6"
server_key_iv: "75fed6adac2b96f47aae9c8735290240"
server_key_version: "encryptionkey@0"
I20260925 15:10:08.886725 18750 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 15:10:08.889734 18765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:08.890925 18750 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:08.891081 18750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "30d2734b82834aa798d17851baf983ea"
format_stamp: "Formatted at 2026-09-25 15:10:08 on dist-test-slave-64fx"
server_key: "a8f7ef904d41d70953a150c6456cffb6"
server_key_iv: "75fed6adac2b96f47aae9c8735290240"
server_key_version: "encryptionkey@0"
I20260925 15:10:08.891252 18750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:08.903646 18750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:08.907282 18750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:08.907552 18750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:08.924906 18750 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33849
I20260925 15:10:08.924932 18816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33849 every 8 connection(s)
I20260925 15:10:08.926280 18750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:08.929678 18817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:08.935874 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18750
I20260925 15:10:08.936086 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:10:08.936697 4919 external_mini_cluster.cc:1428] Setting key 82ddc5ba676bfd23798b7aec6f46d59c
I20260925 15:10:08.939733 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap starting.
I20260925 15:10:08.943724 18817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:08.944641 18817 log.cc:824] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:08.947695 18817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: No bootstrap required, opened a new log
Sep 25 15:10:08 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:08.958504 18817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:08.958943 18817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:08.959056 18817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Initialized, Role: FOLLOWER
I20260925 15:10:08.959689 18817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:08.960048 18817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:08.960217 18817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:08.960479 18817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:08.964092 18817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:08.964640 18817 leader_election.cc:302] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d2734b82834aa798d17851baf983ea; no voters:
I20260925 15:10:08.965271 18817 leader_election.cc:288] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:10:08.965381 18820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:08.938511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.4.205.254:33849)
0925 15:10:08.939238 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:08.939252 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:08.939302 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:10:08.941815 (+ 2513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:08.941848 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:08.942320 (+ 472us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:08.942719 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:08.945256 (+ 2537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:08.946951 (+ 1695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:08.948579 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:08.949581 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:08.949774 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:08.955975 (+ 6201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:08.955986 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:08.956023 (+ 37us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:08.956072 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:08.960710 (+ 4638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:08.961378 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:08.961388 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:08.961396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:08.961627 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:08.962085 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:08.962097 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:08.962101 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:08.962899 (+ 798us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:08.963227 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:08.963616 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 15:10:08.964002 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":305,"threads_started":1}
I20260925 15:10:08.965381 18822 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:08.967315 18817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:08.970196 18822 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 LEADER]: Becoming Leader. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Running, Role: LEADER
I20260925 15:10:08.970811 18822 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:08.973511 18823 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:08.973876 18823 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:08.974438 18824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d2734b82834aa798d17851baf983ea. Latest consensus state: current_term: 1 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:08.974758 18824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:08.979400 18834 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:08.984509 18834 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:08.994128 18834 catalog_manager.cc:1347] Generated new cluster ID: 1349f125078b48489a9d6545a5a46ab0
I20260925 15:10:08.994212 18834 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:09.012348 18834 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:10:09.013687 18834 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:09.022985 18834 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Generated new TSK 0
I20260925 15:10:09.023970 18834 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:10:09.088495 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:09.401969 18843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:09.402211 18843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:09.402400 18843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:09.409257 18843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:09.409350 18843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:09.409518 18843 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:10:09.419189 18843 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:09.420387 18843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:09.421770 18843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:09.434192 18848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:09.434258 18849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:09.434221 18851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:09.435178 18843 server_base.cc:1034] running on GCE node
I20260925 15:10:09.435674 18843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:09.436497 18843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:09.437700 18843 hybrid_clock.cc:648] HybridClock initialized: now 1790349009437670 us; error 38 us; skew 500 ppm
I20260925 15:10:09.438014 18843 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:09 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:09.440915 18843 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:10:09.442360 18843 webserver.cc:460] Webserver started at http://127.4.205.193:45221/ using document root <none> and password file <none>
I20260925 15:10:09.443035 18843 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:09.443171 18843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:09.443456 18843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:09.446303 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "a2d69e610cb84134a35f845a16d3aa63"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "657a965e007347c96ea417423e296118"
server_key_iv: "96f39515afb0661bac709691fa8c6345"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.447021 18843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "a2d69e610cb84134a35f845a16d3aa63"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "657a965e007347c96ea417423e296118"
server_key_iv: "96f39515afb0661bac709691fa8c6345"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.451788 18843 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:10:09.455092 18858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.456224 18843 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:10:09.456393 18843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "a2d69e610cb84134a35f845a16d3aa63"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "657a965e007347c96ea417423e296118"
server_key_iv: "96f39515afb0661bac709691fa8c6345"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.456581 18843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:09.470455 18843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:09.474162 18843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:09.474421 18843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:09.475271 18843 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:09.476486 18843 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:09.476589 18843 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.476727 18843 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:09.476801 18843 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.508931 18843 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:38727
I20260925 15:10:09.508973 18970 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:38727 every 8 connection(s)
I20260925 15:10:09.510234 18843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:10:09.519026 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18843
I20260925 15:10:09.519264 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:10:09.520066 4919 external_mini_cluster.cc:1428] Setting key 4f50bc742a596de3448e3d6814034b32
Sep 25 15:10:09 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:09.531822 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:09.512696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:55407 (local address 127.4.205.254:33849)
0925 15:10:09.513125 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:09.513129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:09.514049 (+ 920us) server_negotiation.cc:407] Connection header received
0925 15:10:09.515809 (+ 1760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:09.515814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:09.515900 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:09.516016 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:09.517386 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:09.518184 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:09.520503 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:09.521055 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:09.521189 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:09.525039 (+ 3850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:09.525047 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:09.525056 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:09.525097 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:09.528726 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:09.529493 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:09.529501 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:09.529508 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:09.529679 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:09.530072 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:09.530079 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:09.530086 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:09.530398 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:09.530506 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:09.531400 (+ 894us) server_negotiation.cc:299] Negotiation successful
0925 15:10:09.531583 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":210,"threads_started":1}
I20260925 15:10:09.533250 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:09.513233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:55407)
0925 15:10:09.513891 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:09.513935 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:09.515519 (+ 1584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:09.516239 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:09.516254 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:09.516701 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:09.517203 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:09.517218 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:09.518337 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:09.518345 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:09.520366 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:09.520380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:09.521287 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:09.521292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:09.521475 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:09.522234 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:09.522263 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:09.524822 (+ 2559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:09.528972 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:09.528978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:09.528989 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:09.529368 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:09.529782 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:09.529785 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:09.529788 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:09.529962 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:09.530538 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:09.530548 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:09.531007 (+ 459us) client_negotiation.cc:769] Sending connection context
0925 15:10:09.531329 (+ 322us) client_negotiation.cc:240] Negotiation successful
0925 15:10:09.531885 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":287,"threads_started":1}
I20260925 15:10:09.534729 18971 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:09.535029 18971 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:09.535739 18971 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:09.537725 18782 ts_manager.cc:194] Registered new tserver with Master: a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727)
I20260925 15:10:09.539119 18782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:55407
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:10:09.571969 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:09.881754 18977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:09.882002 18977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:09.882143 18977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:09.888607 18977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:09.888684 18977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:09.888804 18977 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:10:09.897377 18977 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:09.898407 18977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:09.899508 18977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:09.911825 18983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:09.911862 18982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:09.912446 18985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:09.912719 18977 server_base.cc:1034] running on GCE node
I20260925 15:10:09.913307 18977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:09.914292 18977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:09.915498 18977 hybrid_clock.cc:648] HybridClock initialized: now 1790349009915443 us; error 46 us; skew 500 ppm
I20260925 15:10:09.915782 18977 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:09 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:09.918596 18977 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:10:09.919796 18977 webserver.cc:460] Webserver started at http://127.4.205.194:35255/ using document root <none> and password file <none>
I20260925 15:10:09.920395 18977 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:09.920521 18977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:09.920806 18977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:09.923529 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "536d2e699ddf22136a1c28f226b24436"
server_key_iv: "3f5d1c0191c13e49a119f5f328d730b3"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.924254 18977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "536d2e699ddf22136a1c28f226b24436"
server_key_iv: "3f5d1c0191c13e49a119f5f328d730b3"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.928994 18977 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260925 15:10:09.932314 18992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.933641 18977 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:10:09.933794 18977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "536d2e699ddf22136a1c28f226b24436"
server_key_iv: "3f5d1c0191c13e49a119f5f328d730b3"
server_key_version: "encryptionkey@0"
I20260925 15:10:09.933979 18977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:09.953326 18977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:09.957019 18977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:09.957278 18977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:09.958181 18977 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:09.959409 18977 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:09.959522 18977 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.959656 18977 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:09.959721 18977 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:09.991328 18977 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:37485
I20260925 15:10:09.991377 19104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:37485 every 8 connection(s)
I20260925 15:10:09.992588 18977 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:10:09.995718 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 18977
I20260925 15:10:09.995909 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:10:09.996438 4919 external_mini_cluster.cc:1428] Setting key 79470443b7f50839403602d80c986e1c
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349009, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:10.013041 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:09.995026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:56399 (local address 127.4.205.254:33849)
0925 15:10:09.995188 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:09.995193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:09.996607 (+ 1414us) server_negotiation.cc:407] Connection header received
0925 15:10:09.998926 (+ 2319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:09.998932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:09.999017 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:09.999125 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.000500 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.001715 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.003181 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.003902 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.004038 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.007578 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.007582 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.007586 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.007615 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.010205 (+ 2590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.010887 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.010895 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.010899 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.011038 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.011443 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.011449 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.011453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.011694 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.011856 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.012478 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.012701 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:10:10.013988 19107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:09.995507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:56399)
0925 15:10:09.996435 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:09.996482 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:09.998685 (+ 2203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:09.999305 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:09.999313 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:09.999777 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:10.000315 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.000331 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.001840 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.001844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.003040 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.003052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.004087 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.004092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.004279 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.004927 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:10.004954 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:10.007383 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:10.010345 (+ 2962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:10.010354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:10.010369 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:10.010765 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:10.011146 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:10.011152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:10.011157 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:10.011336 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:10.011816 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:10.011823 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:10.012141 (+ 318us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.012391 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.012874 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":493,"threads_started":1}
I20260925 15:10:10.015463 19105 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:10.015815 19105 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:10.016767 19105 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:10.018379 18782 ts_manager.cc:194] Registered new tserver with Master: 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:10:10.019141 18782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:56399
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:10:10.044750 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:10.353616 19110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:10.353843 19110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:10.353984 19110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:10.360625 19110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:10.360730 19110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:10.360916 19110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:10:10.369961 19110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:10.371122 19110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:10.372364 19110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:10.384884 19118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:10.384866 19115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:10.384887 19116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:10.390765 19110 server_base.cc:1034] running on GCE node
I20260925 15:10:10.391261 19110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:10.392055 19110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:10.393263 19110 hybrid_clock.cc:648] HybridClock initialized: now 1790349010393225 us; error 51 us; skew 500 ppm
I20260925 15:10:10.393533 19110 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:10.397465 19110 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:10:10.399272 19110 webserver.cc:460] Webserver started at http://127.4.205.195:34181/ using document root <none> and password file <none>
I20260925 15:10:10.399984 19110 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:10.400110 19110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:10.400439 19110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:10.404043 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
format_stamp: "Formatted at 2026-09-25 15:10:10 on dist-test-slave-64fx"
server_key: "da076aad7093a817f6409a6f85a6e5af"
server_key_iv: "1dfb9dbec2eb33a3610fffde241484fb"
server_key_version: "encryptionkey@0"
I20260925 15:10:10.404760 19110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
format_stamp: "Formatted at 2026-09-25 15:10:10 on dist-test-slave-64fx"
server_key: "da076aad7093a817f6409a6f85a6e5af"
server_key_iv: "1dfb9dbec2eb33a3610fffde241484fb"
server_key_version: "encryptionkey@0"
I20260925 15:10:10.409528 19110 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 15:10:10.412619 19125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:10.413973 19110 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 15:10:10.414144 19110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
format_stamp: "Formatted at 2026-09-25 15:10:10 on dist-test-slave-64fx"
server_key: "da076aad7093a817f6409a6f85a6e5af"
server_key_iv: "1dfb9dbec2eb33a3610fffde241484fb"
server_key_version: "encryptionkey@0"
I20260925 15:10:10.414327 19110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:10.440999 19110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:10.444779 19110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:10.445068 19110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:10.446058 19110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:10.447260 19110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:10.447360 19110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:10.447501 19110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:10.447569 19110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:10.478801 19110 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:45485
I20260925 15:10:10.478842 19237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:45485 every 8 connection(s)
I20260925 15:10:10.479982 19110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:10:10.484285 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19110
I20260925 15:10:10.484498 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:10:10.484927 4919 external_mini_cluster.cc:1428] Setting key f02d40875ab9823ddc6ab045af8ccf85
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349010, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:10.499013 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.482476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:34539 (local address 127.4.205.254:33849)
0925 15:10:10.482652 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.482656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.483790 (+ 1134us) server_negotiation.cc:407] Connection header received
0925 15:10:10.485769 (+ 1979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.485774 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.485871 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.485979 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.487505 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.488359 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.489859 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.490310 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.490412 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.493415 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.493420 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.493423 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.493455 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.496322 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.496929 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.496938 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.496941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.497049 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.497398 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.497400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.497402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.497731 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.497839 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.498432 (+ 593us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.498635 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:10:10.500177 19240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.482956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:34539)
0925 15:10:10.483638 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.483683 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.485464 (+ 1781us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.486217 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.486224 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.486758 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:10.487319 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.487342 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.488482 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.488486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.489734 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.489745 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.490481 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.490485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.490658 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.491227 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:10.491249 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:10.493246 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:10.496433 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:10.496440 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:10.496450 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:10.496837 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:10.497135 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:10.497138 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:10.497140 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:10.497294 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:10.497830 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:10.497835 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:10.498115 (+ 280us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.498337 (+ 222us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.498848 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":265,"threads_started":1}
I20260925 15:10:10.501470 19238 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:10.501799 19238 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:10.502463 19238 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:10.503751 18782 ts_manager.cc:194] Registered new tserver with Master: 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485)
I20260925 15:10:10.504467 18782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:34539
I20260925 15:10:10.512118 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:10:10.528440 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.516879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.4.205.254:33849)
0925 15:10:10.517048 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.517052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.517509 (+ 457us) server_negotiation.cc:407] Connection header received
0925 15:10:10.517706 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.517709 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.517766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.517853 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.519151 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.519835 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.521074 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.521494 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.521673 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.523822 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.523827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.523830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.523858 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.526410 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.526995 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.526999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.527001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.527119 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.527390 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.527392 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.527394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.527694 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.527860 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.528094 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.528238 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:10:10.531781 18782 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41312:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:10:10.538545 18782 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:10:10.541858 18971 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:10.559598 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.550155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41986 (local address 127.4.205.195:45485)
0925 15:10:10.551246 (+ 1091us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.551252 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.551275 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:10:10.551392 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.551401 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.551659 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.551898 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:10.553391 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.554577 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.557256 (+ 2679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.558469 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.558601 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.558855 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.559095 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.559255 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":894,"thread_start_us":718,"threads_started":1}
I20260925 15:10:10.559629 19250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.549959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45485 (local address 127.0.0.1:41986)
0925 15:10:10.550776 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.550818 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.550986 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.552276 (+ 1290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.552283 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.552318 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:10.552640 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.552655 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.554703 (+ 2048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.554711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.557081 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.557094 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.558645 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.558650 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.558849 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.558906 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.559032 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.559211 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":425,"threads_started":1}
I20260925 15:10:10.560344 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.551263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37485 (local address 127.0.0.1:35644)
0925 15:10:10.551764 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.551799 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.551912 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.552979 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.552986 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.553021 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:10.553617 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.553633 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.555841 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.555848 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.557866 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.557879 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.559621 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.559629 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.559791 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.559852 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.559937 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.560071 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260925 15:10:10.561751 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.551517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35644 (local address 127.4.205.194:37485)
0925 15:10:10.552388 (+ 871us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.552396 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.552424 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:10:10.552517 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.552527 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.552756 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.553865 (+ 1109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:10.554151 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.555701 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.558032 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.559383 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.560970 (+ 1587us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.561155 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.561216 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.561422 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":705,"thread_start_us":590,"threads_started":1}
I20260925 15:10:10.563400 19254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.551821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38727 (local address 127.0.0.1:44584)
0925 15:10:10.555162 (+ 3341us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.555178 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.555290 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.555781 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.555787 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.555817 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:10.556916 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.556932 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.558523 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.558530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.560742 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.560757 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.562674 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.562682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.562852 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.562881 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.562978 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.563098 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3204,"thread_start_us":3031,"threads_started":1}
I20260925 15:10:10.563472 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.552386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44584 (local address 127.4.205.193:38727)
0925 15:10:10.553013 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.553022 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.555279 (+ 2257us) server_negotiation.cc:407] Connection header received
0925 15:10:10.555411 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.555418 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.555593 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.555772 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:10.557122 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.558383 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.560918 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.562418 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.562550 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.562758 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.563030 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.563189 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":299,"threads_started":1}
I20260925 15:10:10.563652 19173 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 (DEFAULT_TABLE table=test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:10.564692 19040 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 (DEFAULT_TABLE table=test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:10.565140 19173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4bcacc28e48b8a19183524fabc6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:10.566241 19040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4bcacc28e48b8a19183524fabc6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:10.566658 18906 tablet_service.cc:1467] Processing CreateTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 (DEFAULT_TABLE table=test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:10.567878 18906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c9e4bcacc28e48b8a19183524fabc6f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:10.577023 19256 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Bootstrap starting.
I20260925 15:10:10.580813 19256 tablet_bootstrap.cc:654] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:10.581629 19257 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Bootstrap starting.
I20260925 15:10:10.582155 19256 log.cc:824] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:10.583559 19258 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Bootstrap starting.
I20260925 15:10:10.584961 19256 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: No bootstrap required, opened a new log
I20260925 15:10:10.585101 19257 tablet_bootstrap.cc:654] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:10.585266 19256 ts_tablet_manager.cc:1397] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20260925 15:10:10.586226 19257 log.cc:824] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:10.586755 19258 tablet_bootstrap.cc:654] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:10.587913 19258 log.cc:824] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:10.588881 19257 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: No bootstrap required, opened a new log
I20260925 15:10:10.589129 19257 ts_tablet_manager.cc:1397] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 15:10:10.590204 19258 tablet_bootstrap.cc:492] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: No bootstrap required, opened a new log
I20260925 15:10:10.590432 19258 ts_tablet_manager.cc:1397] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 15:10:10.594959 19256 raft_consensus.cc:348] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.595359 19256 raft_consensus.cc:374] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:10.595518 19256 raft_consensus.cc:729] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576c62eea15646e8b6cdd2e8c8b6f958, State: Initialized, Role: FOLLOWER
I20260925 15:10:10.596170 19256 consensus_queue.cc:260] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.597193 19105 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:10.597759 19256 ts_tablet_manager.cc:1428] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260925 15:10:10.597720 19258 raft_consensus.cc:348] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.598114 19258 raft_consensus.cc:374] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:10.598232 19258 raft_consensus.cc:729] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d69e610cb84134a35f845a16d3aa63, State: Initialized, Role: FOLLOWER
I20260925 15:10:10.598730 19258 consensus_queue.cc:260] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.598884 19257 raft_consensus.cc:348] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.599352 19257 raft_consensus.cc:374] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:10.599496 19257 raft_consensus.cc:729] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f5863e48ea4567a58bfc6f25fb9c1f, State: Initialized, Role: FOLLOWER
I20260925 15:10:10.600380 19258 ts_tablet_manager.cc:1428] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260925 15:10:10.600234 19257 consensus_queue.cc:260] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.602734 19257 ts_tablet_manager.cc:1428] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Time spent starting tablet: real 0.014s user 0.008s sys 0.006s
I20260925 15:10:10.603137 19238 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:10.612957 19263 raft_consensus.cc:482] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:10.613180 19263 raft_consensus.cc:504] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.614691 19263 leader_election.cc:288] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485), 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
I20260925 15:10:10.621493 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.615087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:54333 (local address 127.4.205.195:45485)
0925 15:10:10.615302 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.615312 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.615741 (+ 429us) server_negotiation.cc:407] Connection header received
0925 15:10:10.615947 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.615956 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.616044 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.616148 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:10.616806 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.617889 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.619827 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.620552 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.620727 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.620801 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.621075 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.621197 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:10:10.621512 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.616005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:43233 (local address 127.4.205.194:37485)
0925 15:10:10.616201 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.616207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.616441 (+ 234us) server_negotiation.cc:407] Connection header received
0925 15:10:10.616573 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.616578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.616654 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.616752 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:10.617320 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.618114 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.619882 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.620614 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.620773 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.620857 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.621102 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.621221 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:10:10.621752 19265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.614990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45485 (local address 127.4.205.193:54333)
0925 15:10:10.615642 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.615658 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.615811 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.616247 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.616260 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.616281 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:10.616669 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.616683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.618019 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.618028 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.619678 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.619689 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.620766 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.620772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.620923 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.620948 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.621053 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.621181 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":42,"thread_start_us":406,"threads_started":1}
I20260925 15:10:10.621788 19266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.615777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37485 (local address 127.4.205.193:43233)
0925 15:10:10.616348 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:10.616364 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:10.616466 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:10.616790 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:10.616795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:10.616816 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:10.617150 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.617161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.618239 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.618249 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.619738 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:10.619749 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.620819 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.620825 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:10.620963 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.620984 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:10:10.621079 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:10:10.621195 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":17,"thread_start_us":273,"threads_started":1}
I20260925 15:10:10.622675 19060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4bcacc28e48b8a19183524fabc6f9" candidate_uuid: "a2d69e610cb84134a35f845a16d3aa63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" is_pre_election: true
I20260925 15:10:10.622712 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4bcacc28e48b8a19183524fabc6f9" candidate_uuid: "a2d69e610cb84134a35f845a16d3aa63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" is_pre_election: true
I20260925 15:10:10.623116 19060 raft_consensus.cc:2454] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2d69e610cb84134a35f845a16d3aa63 in term 0.
I20260925 15:10:10.623132 19193 raft_consensus.cc:2454] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2d69e610cb84134a35f845a16d3aa63 in term 0.
I20260925 15:10:10.623714 18859 leader_election.cc:302] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 576c62eea15646e8b6cdd2e8c8b6f958, a2d69e610cb84134a35f845a16d3aa63; no voters:
I20260925 15:10:10.624080 19263 raft_consensus.cc:2781] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:10:10.624192 19263 raft_consensus.cc:482] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:10:10.624336 19263 raft_consensus.cc:3037] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:10.626832 19263 raft_consensus.cc:504] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.627393 19263 leader_election.cc:288] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [CANDIDATE]: Term 1 election: Requested vote from peers 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485), 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
I20260925 15:10:10.627794 19193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4bcacc28e48b8a19183524fabc6f9" candidate_uuid: "a2d69e610cb84134a35f845a16d3aa63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
I20260925 15:10:10.627877 19060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c9e4bcacc28e48b8a19183524fabc6f9" candidate_uuid: "a2d69e610cb84134a35f845a16d3aa63" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
I20260925 15:10:10.627969 19193 raft_consensus.cc:3037] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:10.628093 19060 raft_consensus.cc:3037] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:10.630163 19193 raft_consensus.cc:2454] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2d69e610cb84134a35f845a16d3aa63 in term 1.
I20260925 15:10:10.630295 19060 raft_consensus.cc:2454] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2d69e610cb84134a35f845a16d3aa63 in term 1.
I20260925 15:10:10.630535 18859 leader_election.cc:302] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f5863e48ea4567a58bfc6f25fb9c1f, a2d69e610cb84134a35f845a16d3aa63; no voters:
I20260925 15:10:10.630800 19263 raft_consensus.cc:2781] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:10.631196 19263 raft_consensus.cc:686] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 1 LEADER]: Becoming Leader. State: Replica: a2d69e610cb84134a35f845a16d3aa63, State: Running, Role: LEADER
I20260925 15:10:10.631714 19263 consensus_queue.cc:237] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } }
I20260925 15:10:10.635720 18782 catalog_manager.cc:5495] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "a2d69e610cb84134a35f845a16d3aa63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } health_report { overall_health: HEALTHY } } }
I20260925 15:10:10.656848 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.651576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44586 (local address 127.4.205.193:38727)
0925 15:10:10.651770 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.651775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.651876 (+ 101us) server_negotiation.cc:407] Connection header received
0925 15:10:10.652151 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.652155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.652225 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.652386 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:10.652752 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.653447 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.654884 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.655310 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.655439 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.655881 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:10.656316 (+ 435us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:10:10.656482 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.656531 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.656604 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:10:10.664765 19193 raft_consensus.cc:1261] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Refusing update from remote peer a2d69e610cb84134a35f845a16d3aa63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:10.664772 19060 raft_consensus.cc:1261] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Refusing update from remote peer a2d69e610cb84134a35f845a16d3aa63: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:10.665711 19267 consensus_queue.cc:1035] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:10.666069 19263 consensus_queue.cc:1035] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:10.690366 18781 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:10.690685 18781 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41312:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:10.694622 18781 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Sending DeleteTablet for 3 replicas of tablet c9e4bcacc28e48b8a19183524fabc6f9
I20260925 15:10:10.695884 19173 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:10 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:41986
I20260925 15:10:10.696018 19040 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:10 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:35644
I20260925 15:10:10.696031 18906 tablet_service.cc:1514] Processing DeleteTablet for tablet c9e4bcacc28e48b8a19183524fabc6f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:10 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:44584
I20260925 15:10:10.696772 19280 tablet_replica.cc:330] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: stopping tablet replica
I20260925 15:10:10.697033 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18843
I20260925 15:10:10.697325 19280 raft_consensus.cc:2229] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:10.698356 19280 raft_consensus.cc:2258] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:10.701295 19280 ts_tablet_manager.cc:1905] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:10.710088 19282 tablet_replica.cc:330] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: stopping tablet replica
I20260925 15:10:10.710549 19282 raft_consensus.cc:2229] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:10.711064 19282 raft_consensus.cc:2258] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:10.713338 19282 ts_tablet_manager.cc:1905] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:10.714103 19280 ts_tablet_manager.cc:1918] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:10.714206 19281 tablet_replica.cc:330] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: stopping tablet replica
I20260925 15:10:10.714299 19280 log.cc:1196] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/c9e4bcacc28e48b8a19183524fabc6f9
I20260925 15:10:10.714831 19280 ts_tablet_manager.cc:1939] T c9e4bcacc28e48b8a19183524fabc6f9 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting consensus metadata
I20260925 15:10:10.715765 19281 raft_consensus.cc:2229] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:10:10.716441 19281 raft_consensus.cc:2258] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:10.716657 18766 catalog_manager.cc:4841] TS 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485): tablet c9e4bcacc28e48b8a19183524fabc6f9 (table test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]) successfully deleted
I20260925 15:10:10.718917 19281 ts_tablet_manager.cc:1905] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:10.722715 19282 ts_tablet_manager.cc:1918] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:10.722869 19282 log.cc:1196] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/c9e4bcacc28e48b8a19183524fabc6f9
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:10:10.723389 19282 ts_tablet_manager.cc:1939] T c9e4bcacc28e48b8a19183524fabc6f9 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting consensus metadata
I20260925 15:10:10.725147 18766 catalog_manager.cc:4841] TS 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485): tablet c9e4bcacc28e48b8a19183524fabc6f9 (table test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]) successfully deleted
I20260925 15:10:10.726334 19281 ts_tablet_manager.cc:1918] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:10.726509 19281 log.cc:1196] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/c9e4bcacc28e48b8a19183524fabc6f9
I20260925 15:10:10.727017 19281 ts_tablet_manager.cc:1939] T c9e4bcacc28e48b8a19183524fabc6f9 P a2d69e610cb84134a35f845a16d3aa63: Deleting consensus metadata
I20260925 15:10:10.728608 18766 catalog_manager.cc:4841] TS a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727): tablet c9e4bcacc28e48b8a19183524fabc6f9 (table test-table [id=bb4ecb32c57a4793a54b5f9e71af0d05]) successfully deleted
I20260925 15:10:10.730839 19255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.697775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44590 (local address 127.4.205.193:38727)
0925 15:10:10.697948 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.697955 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.710791 (+ 12836us) server_negotiation.cc:407] Connection header received
0925 15:10:10.711704 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.711711 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.711805 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.711876 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.713916 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.715146 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.718072 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.718777 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.718865 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.724584 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.724599 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.724619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.724668 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.727975 (+ 3307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.728699 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.728712 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.728715 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.728892 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.729270 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.729276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.729280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.729679 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.729977 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.730233 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.730449 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:10:10.731577 18966 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:10.829944 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18843
I20260925 15:10:10.848336 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18977
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:10:10.862244 19253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.849059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35648 (local address 127.4.205.194:37485)
0925 15:10:10.849229 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.849237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.849340 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:10:10.849597 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.849603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.849673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.849792 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.850978 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.851684 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.852946 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.853344 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.853477 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.857436 (+ 3959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.857448 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.857462 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.857493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.859912 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.860492 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.860498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.860501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.860610 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.860921 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.860924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.860926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.861242 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.861486 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.861853 (+ 367us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.862002 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:10:10.862861 19100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:10.956090 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18977
I20260925 15:10:10.972304 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19110
Sep 25 15:10:10 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349008, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:10:10.986239 19251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:10.973043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41994 (local address 127.4.205.195:45485)
0925 15:10:10.973211 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:10.973216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:10.973233 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:10:10.973476 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:10.973482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:10.973638 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:10.973725 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:10.975003 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.975704 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.976932 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:10.977294 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:10.977388 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:10.981478 (+ 4090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:10.981491 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:10.981505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:10.981533 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:10.984061 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.984623 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.984630 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.984632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.984745 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:10.985044 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:10.985047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:10.985049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:10.985368 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:10.985629 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:10.985865 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:10:10.986012 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:10:10.986855 19233 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:11.075016 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19110
I20260925 15:10:11.091251 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 18750
I20260925 15:10:11.092423 18812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:11.200217 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 18750
I20260925 15:10:11.221762 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--webserver_interface=127.4.205.254
--webserver_port=39781
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:11.532800 19288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:11.533057 19288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:11.533237 19288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:11.539978 19288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:11.540088 19288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:11.540155 19288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:11.540236 19288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:11.540284 19288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:11.549299 19288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=39781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:11.550453 19288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:11.551561 19288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:11.572865 19295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:11.572911 19294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:11.572818 19297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:11.573618 19288 server_base.cc:1034] running on GCE node
I20260925 15:10:11.574417 19288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:11.575879 19288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:11.577111 19288 hybrid_clock.cc:648] HybridClock initialized: now 1790349011577048 us; error 43 us; skew 500 ppm
I20260925 15:10:11.577482 19288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:11 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349011, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:11.581007 19288 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:11.582587 19288 webserver.cc:460] Webserver started at http://127.4.205.254:39781/ using document root <none> and password file <none>
I20260925 15:10:11.583240 19288 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:11.583349 19288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:11.588649 19288 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:10:11.591245 19304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:11.592468 19288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:10:11.592604 19288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "30d2734b82834aa798d17851baf983ea"
format_stamp: "Formatted at 2026-09-25 15:10:08 on dist-test-slave-64fx"
server_key: "a8f7ef904d41d70953a150c6456cffb6"
server_key_iv: "75fed6adac2b96f47aae9c8735290240"
server_key_version: "encryptionkey@0"
I20260925 15:10:11.593616 19288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:11.614006 19288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:11.618145 19288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:11.618356 19288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:11.636251 19288 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33849
I20260925 15:10:11.636281 19355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33849 every 8 connection(s)
I20260925 15:10:11.637677 19288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:11.641986 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19288
I20260925 15:10:11.642759 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:38727
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=45221
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:11.644088 19356 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:10:11.647238 19356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap starting.
I20260925 15:10:11.666146 19356 log.cc:824] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:11.678373 19356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:10:11.678900 19356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap complete.
I20260925 15:10:11.687986 19356 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:11.689005 19356 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Initialized, Role: FOLLOWER
I20260925 15:10:11.689661 19356 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:11.689939 19356 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:11.690066 19356 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:11.690243 19356 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:10:11.693971 19356 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:11.694458 19356 leader_election.cc:302] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d2734b82834aa798d17851baf983ea; no voters:
I20260925 15:10:11.695127 19356 leader_election.cc:288] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:10:11.695257 19360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:10:11.696544 19360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 LEADER]: Becoming Leader. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Running, Role: LEADER
I20260925 15:10:11.697114 19360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:11.697577 19356 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:11.699584 19361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:11.699980 19361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:11.703778 19362 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d2734b82834aa798d17851baf983ea. Latest consensus state: current_term: 2 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:11.704116 19362 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:11.705613 19366 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:11.714097 19366 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:11.715658 19366 catalog_manager.cc:1259] Loaded cluster ID: 1349f125078b48489a9d6545a5a46ab0
I20260925 15:10:11.715803 19366 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:11.718698 19366 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:10:11.735965 19366 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:10:11.751289 19366 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:11.752959 19366 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Loaded TSK: 0
I20260925 15:10:11.754386 19366 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:10:11.975230 19358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:11.975507 19358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:11.975660 19358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:11.982635 19358 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:11.982769 19358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:11.982926 19358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:10:11.991801 19358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:38727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=45221
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:11.992949 19358 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:11.994307 19358 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:12.008576 19384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.008621 19383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.008978 19386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:12.009744 19358 server_base.cc:1034] running on GCE node
I20260925 15:10:12.010308 19358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:12.011260 19358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:12.012471 19358 hybrid_clock.cc:648] HybridClock initialized: now 1790349012012395 us; error 71 us; skew 500 ppm
I20260925 15:10:12.012769 19358 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:12.016093 19358 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:10:12.017511 19358 webserver.cc:460] Webserver started at http://127.4.205.193:45221/ using document root <none> and password file <none>
I20260925 15:10:12.018158 19358 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:12.018285 19358 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:12.023382 19358 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.002s
I20260925 15:10:12.025965 19393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.027304 19358 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:10:12.027458 19358 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "a2d69e610cb84134a35f845a16d3aa63"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "657a965e007347c96ea417423e296118"
server_key_iv: "96f39515afb0661bac709691fa8c6345"
server_key_version: "encryptionkey@0"
I20260925 15:10:12.028357 19358 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:12.041855 19358 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:12.045853 19358 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:12.046103 19358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:12.047050 19358 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:12.048331 19358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:12.048424 19358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.048566 19358 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:12.048661 19358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.081076 19358 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:38727
I20260925 15:10:12.081171 19505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:38727 every 8 connection(s)
I20260925 15:10:12.082540 19358 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:10:12.082995 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19358
I20260925 15:10:12.083755 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:37485
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=35255
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:12.105070 19509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.085694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:56293)
0925 15:10:12.086455 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.086499 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.088066 (+ 1567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.089298 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.089310 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.089909 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:12.090398 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.090414 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.092065 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.092069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.093260 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.093271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.094347 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.094354 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.094618 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.095262 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:12.095283 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:12.097521 (+ 2238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:12.101005 (+ 3484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.101016 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.101030 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.101438 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.101886 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.101891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.101896 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.102083 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.102803 (+ 720us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:12.102810 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:12.103116 (+ 306us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.103378 (+ 262us) client_negotiation.cc:240] Negotiation successful
0925 15:10:12.103869 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":308,"threads_started":1}
I20260925 15:10:12.105067 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.085382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:56293 (local address 127.4.205.254:33849)
0925 15:10:12.086120 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.086138 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.086633 (+ 495us) server_negotiation.cc:407] Connection header received
0925 15:10:12.088552 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.088562 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.088842 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.089173 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:12.090642 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.091893 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.093423 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.094121 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.094316 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.097842 (+ 3526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:12.097853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:12.097877 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:12.097920 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:12.100858 (+ 2938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.101600 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.101609 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.101623 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.101772 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.102229 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.102234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.102236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.102663 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:12.102864 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.103590 (+ 726us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.103948 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":492,"thread_start_us":340,"threads_started":1}
I20260925 15:10:12.107772 19506 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:12.108063 19506 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:12.108706 19506 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:12.110713 19321 ts_manager.cc:194] Registered new tserver with Master: a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727)
I20260925 15:10:12.112162 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:56293
W20260925 15:10:12.392874 19510 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:12.393177 19510 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:12.393352 19510 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:12.404999 19510 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:12.405125 19510 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:12.405296 19510 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:10:12.414942 19510 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:37485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=35255
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:12.416213 19510 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:12.417446 19510 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:12.430081 19515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.430109 19516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.430150 19518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:12.430768 19510 server_base.cc:1034] running on GCE node
I20260925 15:10:12.431354 19510 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:12.432255 19510 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:12.433481 19510 hybrid_clock.cc:648] HybridClock initialized: now 1790349012433435 us; error 34 us; skew 500 ppm
I20260925 15:10:12.433797 19510 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:12.436586 19510 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:10:12.438014 19510 webserver.cc:460] Webserver started at http://127.4.205.194:35255/ using document root <none> and password file <none>
I20260925 15:10:12.438701 19510 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:12.438851 19510 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:12.443861 19510 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20260925 15:10:12.446452 19525 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.447659 19510 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:10:12.447831 19510 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "536d2e699ddf22136a1c28f226b24436"
server_key_iv: "3f5d1c0191c13e49a119f5f328d730b3"
server_key_version: "encryptionkey@0"
I20260925 15:10:12.448783 19510 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:12.465257 19510 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:12.468875 19510 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:12.469169 19510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:12.470122 19510 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:12.471309 19510 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:12.471443 19510 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.471570 19510 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:12.471653 19510 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.502259 19510 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:37485
I20260925 15:10:12.502339 19637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:37485 every 8 connection(s)
I20260925 15:10:12.503674 19510 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:10:12.505803 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19510
I20260925 15:10:12.506440 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:45485
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=34181
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:12.522696 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.506196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:60041 (local address 127.4.205.254:33849)
0925 15:10:12.506423 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.506429 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.507692 (+ 1263us) server_negotiation.cc:407] Connection header received
0925 15:10:12.509535 (+ 1843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.509541 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.509663 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.509776 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:12.511143 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.511828 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.513341 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.513874 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.514011 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.517494 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:12.517499 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:12.517502 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:12.517532 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:12.519814 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.520462 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.520468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.520473 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.520589 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.520975 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.520980 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.520984 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.521227 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:12.521441 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.522132 (+ 691us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.522356 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 15:10:12.523535 19640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.506814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:60041)
0925 15:10:12.507545 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.507591 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.509265 (+ 1674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.510027 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.510035 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.510471 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:12.510981 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.510997 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.511995 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.511999 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.513190 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.513202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.514080 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.514087 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.514368 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.515052 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:12.515082 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:12.517301 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:12.519939 (+ 2638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.519948 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.519960 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.520341 (+ 381us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.520694 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.520699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.520704 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.520863 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.521415 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:12.521422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:12.521785 (+ 363us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.522046 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:10:12.522496 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":306,"threads_started":1}
I20260925 15:10:12.524950 19638 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:12.525220 19638 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:12.526000 19638 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:12.527374 19321 ts_manager.cc:194] Registered new tserver with Master: 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
I20260925 15:10:12.528622 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:60041
W20260925 15:10:12.814514 19641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:12.814783 19641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:12.814864 19641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:12.822928 19641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:12.823024 19641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:12.823136 19641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:10:12.831807 19641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=34181
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:12.832957 19641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:12.834203 19641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:12.846947 19646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.846918 19647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:12.846868 19649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:12.847543 19641 server_base.cc:1034] running on GCE node
I20260925 15:10:12.848045 19641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:12.848826 19641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:12.850019 19641 hybrid_clock.cc:648] HybridClock initialized: now 1790349012849984 us; error 27 us; skew 500 ppm
I20260925 15:10:12.850296 19641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:12.853384 19641 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:10:12.854705 19641 webserver.cc:460] Webserver started at http://127.4.205.195:34181/ using document root <none> and password file <none>
I20260925 15:10:12.855289 19641 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:12.855391 19641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:12.860303 19641 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:10:12.862751 19656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.863941 19641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:12.864101 19641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
format_stamp: "Formatted at 2026-09-25 15:10:10 on dist-test-slave-64fx"
server_key: "da076aad7093a817f6409a6f85a6e5af"
server_key_iv: "1dfb9dbec2eb33a3610fffde241484fb"
server_key_version: "encryptionkey@0"
I20260925 15:10:12.864971 19641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:12.890690 19641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:12.894574 19641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:12.894836 19641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:12.895639 19641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:12.896840 19641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:12.896955 19641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.897080 19641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:12.897137 19641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:12.928488 19641 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:45485
I20260925 15:10:12.928567 19768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:45485 every 8 connection(s)
I20260925 15:10:12.930030 19641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:10:12.936976 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19641
Sep 25 15:10:12 dist-test-slave-64fx krb5kdc[18737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349012, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:12.949350 19508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.932364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:47031 (local address 127.4.205.254:33849)
0925 15:10:12.932534 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.932539 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.933621 (+ 1082us) server_negotiation.cc:407] Connection header received
0925 15:10:12.935277 (+ 1656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.935282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.935372 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.935467 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:12.936864 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.937660 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.939416 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.939838 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.939961 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.943452 (+ 3491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:12.943457 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:12.943460 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:12.943489 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:12.946559 (+ 3070us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.947233 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.947239 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.947243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.947370 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.947799 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.947809 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.947813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.948071 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:12.948230 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.948836 (+ 606us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.949040 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:10:12.950243 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.932818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:47031)
0925 15:10:12.933469 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.933502 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.935039 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.935674 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.935680 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.936135 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:12.936687 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.936712 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.937862 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.937867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.939258 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.939270 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.940043 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.940050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.940267 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.940892 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:12.940912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:12.943241 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:12.946686 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.946696 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.946710 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.947100 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.947495 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:12.947500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:12.947504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:12.947687 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:12.948211 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:12.948218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:12.948513 (+ 295us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.948759 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 15:10:12.949216 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":253,"threads_started":1}
I20260925 15:10:12.950886 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.937847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.4.205.254:33849)
0925 15:10:12.938300 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.938307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.938577 (+ 270us) server_negotiation.cc:407] Connection header received
0925 15:10:12.938748 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.938753 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.938827 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.938928 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:12.940264 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.941147 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.942601 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.943081 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.943212 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.945614 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:12.945621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:12.945626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:12.945676 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:12.948215 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.948831 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.948838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.948842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.948960 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:12.949294 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:12.949300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:12.949304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:12.949773 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:12.950032 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.950313 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.950501 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":223,"threads_started":1}
I20260925 15:10:12.951694 19769 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33849
I20260925 15:10:12.952008 19769 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:12.952847 19769 heartbeater.cc:511] Master 127.4.205.254:33849 requested a full tablet report, sending...
I20260925 15:10:12.954244 19321 ts_manager.cc:194] Registered new tserver with Master: 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485)
I20260925 15:10:12.954944 19321 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:47031
I20260925 15:10:12.965363 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:10:12.971611 19772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.966198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.4.205.254:33849)
0925 15:10:12.966348 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.966352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.966774 (+ 422us) server_negotiation.cc:407] Connection header received
0925 15:10:12.966947 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.966951 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.967055 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.967160 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:12.967518 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.968216 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.969838 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.970265 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.970352 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.970843 (+ 491us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:12.971226 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:10:12.971321 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.971405 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.971455 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:10:12.972366 19321 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:47574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:10:12.979193 19321 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:10:12.999450 19779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.989476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38727 (local address 127.0.0.1:40618)
0925 15:10:12.990393 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.990458 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.990625 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.991664 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.991671 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.991704 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:12.992043 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.992057 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.994114 (+ 2057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.994119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.996870 (+ 2751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.996883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.998490 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.998496 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.998657 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.998702 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.998809 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 15:10:12.999004 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"mutex_wait_us":80,"thread_start_us":315,"threads_started":1}
I20260925 15:10:12.999879 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.989703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40618 (local address 127.4.205.193:38727)
0925 15:10:12.990502 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.990510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.990538 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:10:12.990947 (+ 409us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.990959 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.991198 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.991378 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:12.992272 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.993961 (+ 1689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.997018 (+ 3057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.998291 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.998617 (+ 326us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.999391 (+ 774us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:12.999457 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:10:12.999605 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":499,"threads_started":1}
I20260925 15:10:13.000061 19781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.990520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45485 (local address 127.0.0.1:41576)
0925 15:10:12.991071 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.991088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.991182 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.993871 (+ 2689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.993875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.993897 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:12.994252 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.994261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.995722 (+ 1461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.995726 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.997661 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.997673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.999304 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.999314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.999515 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.999551 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.999613 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:10:12.999707 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":3,"thread_start_us":318,"threads_started":1}
I20260925 15:10:13.000391 19783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.991167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37485 (local address 127.0.0.1:54408)
0925 15:10:12.991665 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:12.991679 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:12.991771 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:12.993007 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:12.993011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:12.993036 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:12.993345 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.993355 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.995364 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.995370 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.997452 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:12.997465 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.999442 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.999447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:12.999809 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:12.999839 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:10:12.999894 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:10:13.000018 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":22,"thread_start_us":862,"threads_started":1}
I20260925 15:10:13.000871 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.991849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.4.205.194:37485)
0925 15:10:12.992423 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.992443 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.992479 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:10:12.992568 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.992575 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.992799 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.993581 (+ 782us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:12.993805 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.995204 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.997584 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.999088 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.000152 (+ 1064us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.000348 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.000414 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.000561 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":929,"threads_started":1}
I20260925 15:10:13.002857 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:12.990881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.4.205.195:45485)
0925 15:10:12.993423 (+ 2542us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:12.993428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:12.993450 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:10:12.993511 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:12.993515 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:12.993705 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:12.993876 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:12.994401 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.995581 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:12.997795 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:12.999048 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.001880 (+ 2832us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.002081 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.002140 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.002279 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2400,"thread_start_us":2256,"threads_started":1}
I20260925 15:10:13.003610 19441 tablet_service.cc:1467] Processing CreateTablet for tablet 686ea31f724f489ab206105960be1204 (DEFAULT_TABLE table=test-table [id=67b2dbe88ae742518c89bca63736762e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:13.004635 19573 tablet_service.cc:1467] Processing CreateTablet for tablet 686ea31f724f489ab206105960be1204 (DEFAULT_TABLE table=test-table [id=67b2dbe88ae742518c89bca63736762e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:13.005054 19441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ea31f724f489ab206105960be1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:13.005684 19573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ea31f724f489ab206105960be1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:13.006018 19704 tablet_service.cc:1467] Processing CreateTablet for tablet 686ea31f724f489ab206105960be1204 (DEFAULT_TABLE table=test-table [id=67b2dbe88ae742518c89bca63736762e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:13.007195 19704 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 686ea31f724f489ab206105960be1204. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:13.016634 19785 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Bootstrap starting.
I20260925 15:10:13.019465 19785 tablet_bootstrap.cc:654] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:13.019510 19786 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Bootstrap starting.
I20260925 15:10:13.020716 19785 log.cc:824] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:13.022301 19787 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Bootstrap starting.
I20260925 15:10:13.023121 19786 tablet_bootstrap.cc:654] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:13.024499 19786 log.cc:824] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:13.025099 19787 tablet_bootstrap.cc:654] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:13.026264 19787 log.cc:824] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:13.027048 19785 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: No bootstrap required, opened a new log
I20260925 15:10:13.027281 19785 ts_tablet_manager.cc:1397] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.008s
I20260925 15:10:13.027737 19786 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: No bootstrap required, opened a new log
I20260925 15:10:13.027961 19786 ts_tablet_manager.cc:1397] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 15:10:13.028599 19787 tablet_bootstrap.cc:492] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: No bootstrap required, opened a new log
I20260925 15:10:13.028817 19787 ts_tablet_manager.cc:1397] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:10:13.035802 19787 raft_consensus.cc:348] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.036113 19787 raft_consensus.cc:374] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:13.036227 19787 raft_consensus.cc:729] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26f5863e48ea4567a58bfc6f25fb9c1f, State: Initialized, Role: FOLLOWER
I20260925 15:10:13.036764 19787 consensus_queue.cc:260] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.037189 19785 raft_consensus.cc:348] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.037601 19785 raft_consensus.cc:374] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:13.037703 19785 raft_consensus.cc:729] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 576c62eea15646e8b6cdd2e8c8b6f958, State: Initialized, Role: FOLLOWER
I20260925 15:10:13.037796 19786 raft_consensus.cc:348] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.038254 19786 raft_consensus.cc:374] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:13.038307 19769 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:13.038352 19786 raft_consensus.cc:729] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d69e610cb84134a35f845a16d3aa63, State: Initialized, Role: FOLLOWER
I20260925 15:10:13.038245 19785 consensus_queue.cc:260] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.038508 19787 ts_tablet_manager.cc:1428] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20260925 15:10:13.038983 19786 consensus_queue.cc:260] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.042935 19506 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:13.043695 19786 ts_tablet_manager.cc:1428] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260925 15:10:13.044616 19638 heartbeater.cc:503] Master 127.4.205.254:33849 was elected leader, sending a full tablet report...
I20260925 15:10:13.045050 19785 ts_tablet_manager.cc:1428] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Time spent starting tablet: real 0.018s user 0.009s sys 0.010s
I20260925 15:10:13.060015 19791 raft_consensus.cc:482] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:13.060184 19791 raft_consensus.cc:504] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.061353 19791 leader_election.cc:288] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727), 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
I20260925 15:10:13.065721 19792 raft_consensus.cc:482] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:13.066005 19792 raft_consensus.cc:504] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.066851 19771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.061692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38727 (local address 127.4.205.195:57545)
0925 15:10:13.061888 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:13.061907 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:13.062029 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:13.062563 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:13.062568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:13.062597 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:13.062764 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.062772 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.063722 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.063728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.064917 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.064937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.066152 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.066159 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.066331 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.066360 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 15:10:13.066466 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:10:13.066583 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":1}
I20260925 15:10:13.066866 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.061644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:57545 (local address 127.4.205.193:38727)
0925 15:10:13.061803 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.061808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.061981 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:10:13.062164 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.062170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.062256 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.062378 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:13.062900 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.063598 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.065068 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.066003 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.066160 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.066225 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.066513 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.066640 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:10:13.067668 19792 leader_election.cc:288] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727), 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485)
I20260925 15:10:13.068034 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.062223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:43399 (local address 127.4.205.194:37485)
0925 15:10:13.062451 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.062460 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.062838 (+ 378us) server_negotiation.cc:407] Connection header received
0925 15:10:13.063025 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.063030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.063094 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.063183 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:13.063670 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.064265 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.065834 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.066989 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.067135 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.067190 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.067661 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.067810 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:10:13.067937 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d69e610cb84134a35f845a16d3aa63" is_pre_election: true
I20260925 15:10:13.068408 19461 raft_consensus.cc:2454] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f5863e48ea4567a58bfc6f25fb9c1f in term 0.
I20260925 15:10:13.068584 19794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.062082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:37485 (local address 127.4.205.195:43399)
0925 15:10:13.062747 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:13.062760 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:13.062906 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:13.063197 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:13.063201 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:13.063224 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:13.063532 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.063539 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.064431 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.064434 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.065698 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.065708 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.067315 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.067328 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.067484 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.067523 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 15:10:13.068178 (+ 655us) client_negotiation.cc:240] Negotiation successful
0925 15:10:13.068299 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":388,"threads_started":1}
I20260925 15:10:13.069036 19657 leader_election.cc:302] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f5863e48ea4567a58bfc6f25fb9c1f, a2d69e610cb84134a35f845a16d3aa63; no voters:
I20260925 15:10:13.069445 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" is_pre_election: true
I20260925 15:10:13.070004 19593 raft_consensus.cc:2454] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26f5863e48ea4567a58bfc6f25fb9c1f in term 0.
I20260925 15:10:13.070653 19791 raft_consensus.cc:2781] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:10:13.070825 19791 raft_consensus.cc:482] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:10:13.070936 19791 raft_consensus.cc:3037] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:13.073833 19791 raft_consensus.cc:504] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.074461 19791 leader_election.cc:288] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [CANDIDATE]: Term 1 election: Requested vote from peers a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727), 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485)
I20260925 15:10:13.074884 19593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
I20260925 15:10:13.074918 19461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d69e610cb84134a35f845a16d3aa63"
I20260925 15:10:13.075083 19461 raft_consensus.cc:3037] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:13.075098 19593 raft_consensus.cc:3037] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:13.075546 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.068424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53579 (local address 127.4.205.193:38727)
0925 15:10:13.068549 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.068941 (+ 392us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.068962 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:10:13.069010 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.069014 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.069087 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.069165 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:13.070044 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.071191 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.073103 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.074289 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.074450 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.074509 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.075150 (+ 641us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.075261 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:10:13.075975 19795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.067875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:38727 (local address 127.4.205.194:53579)
0925 15:10:13.068405 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:13.068422 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:13.068529 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:13.069361 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:13.069366 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:13.069397 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:13.069887 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.069902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.071312 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.071319 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.072962 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.072972 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.074649 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.074655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.074822 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.074857 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:10:13.074953 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:10:13.075069 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":775,"threads_started":1}
I20260925 15:10:13.076680 19460 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2d69e610cb84134a35f845a16d3aa63" is_pre_election: true
I20260925 15:10:13.077776 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.069311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53737 (local address 127.4.205.195:45485)
0925 15:10:13.069528 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.069533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.071262 (+ 1729us) server_negotiation.cc:407] Connection header received
0925 15:10:13.071405 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.071408 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.071474 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.071565 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:13.072201 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.073288 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.075560 (+ 2272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.076681 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.076851 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.076925 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.077420 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.077547 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 15:10:13.078274 19593 raft_consensus.cc:2454] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f5863e48ea4567a58bfc6f25fb9c1f in term 1.
I20260925 15:10:13.078315 19461 raft_consensus.cc:2454] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26f5863e48ea4567a58bfc6f25fb9c1f in term 1.
I20260925 15:10:13.078359 19796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.069035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45485 (local address 127.4.205.194:53737)
0925 15:10:13.071166 (+ 2131us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:13.071180 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:13.071288 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:13.071586 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:13.071592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:13.071634 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:13.072015 (+ 381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.072042 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.073431 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.073436 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.075401 (+ 1965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:13.075413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.077075 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.077081 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:13.077233 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.077263 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:10:13.077975 (+ 712us) client_negotiation.cc:240] Negotiation successful
0925 15:10:13.078095 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2040,"thread_start_us":1889,"threads_started":1}
I20260925 15:10:13.078673 19657 leader_election.cc:302] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26f5863e48ea4567a58bfc6f25fb9c1f, 576c62eea15646e8b6cdd2e8c8b6f958; no voters:
I20260925 15:10:13.078923 19791 raft_consensus.cc:2781] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:13.079098 19724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "686ea31f724f489ab206105960be1204" candidate_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" is_pre_election: true
I20260925 15:10:13.079428 19791 raft_consensus.cc:686] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 LEADER]: Becoming Leader. State: Replica: 26f5863e48ea4567a58bfc6f25fb9c1f, State: Running, Role: LEADER
I20260925 15:10:13.079901 19791 consensus_queue.cc:237] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } }
I20260925 15:10:13.083765 19526 leader_election.cc:302] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 576c62eea15646e8b6cdd2e8c8b6f958; no voters: 26f5863e48ea4567a58bfc6f25fb9c1f, a2d69e610cb84134a35f845a16d3aa63
I20260925 15:10:13.084113 19792 raft_consensus.cc:2726] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260925 15:10:13.085011 19507 tablet.cc:2369] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:10:13.084892 19320 catalog_manager.cc:5495] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f reported cstate change: term changed from 0 to 1, leader changed from <none> to 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195). New cstate: current_term: 1 leader_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26f5863e48ea4567a58bfc6f25fb9c1f" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 } health_report { overall_health: UNKNOWN } } }
I20260925 15:10:13.100121 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.095594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.4.205.195:45485)
0925 15:10:13.095714 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.095717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.095801 (+ 84us) server_negotiation.cc:407] Connection header received
0925 15:10:13.095949 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.095951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.096055 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.096166 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:13.096445 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.097065 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.098420 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.098853 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.098943 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.099315 (+ 372us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:13.099698 (+ 383us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:10:13.099794 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.099874 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.099937 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:10:13.106634 19461 raft_consensus.cc:1261] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Refusing update from remote peer 26f5863e48ea4567a58bfc6f25fb9c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:13.106807 19593 raft_consensus.cc:1261] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Refusing update from remote peer 26f5863e48ea4567a58bfc6f25fb9c1f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:13.107370 19797 consensus_queue.cc:1035] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2d69e610cb84134a35f845a16d3aa63" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 38727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:13.107656 19791 consensus_queue.cc:1035] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "576c62eea15646e8b6cdd2e8c8b6f958" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 37485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:13.131534 19321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:47574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:13.131803 19321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:47574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:13.135070 19321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Sending DeleteTablet for 3 replicas of tablet 686ea31f724f489ab206105960be1204
I20260925 15:10:13.135943 19441 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ea31f724f489ab206105960be1204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:13 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:40618
I20260925 15:10:13.136204 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19358
I20260925 15:10:13.136417 19573 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ea31f724f489ab206105960be1204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:13 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:54408
I20260925 15:10:13.136520 19704 tablet_service.cc:1514] Processing DeleteTablet for tablet 686ea31f724f489ab206105960be1204 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:13 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:41576
I20260925 15:10:13.137135 19811 tablet_replica.cc:330] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: stopping tablet replica
I20260925 15:10:13.137496 19811 raft_consensus.cc:2229] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:13.137949 19811 raft_consensus.cc:2258] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:13.138310 19810 tablet_replica.cc:330] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: stopping tablet replica
I20260925 15:10:13.138619 19810 raft_consensus.cc:2229] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:13.138964 19810 raft_consensus.cc:2258] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:13.140591 19810 ts_tablet_manager.cc:1905] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:13.141949 19811 ts_tablet_manager.cc:1905] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:13.147135 19810 ts_tablet_manager.cc:1918] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:13.147266 19810 log.cc:1196] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/686ea31f724f489ab206105960be1204
I20260925 15:10:13.147779 19810 ts_tablet_manager.cc:1939] T 686ea31f724f489ab206105960be1204 P a2d69e610cb84134a35f845a16d3aa63: Deleting consensus metadata
I20260925 15:10:13.149381 19305 catalog_manager.cc:4841] TS a2d69e610cb84134a35f845a16d3aa63 (127.4.205.193:38727): tablet 686ea31f724f489ab206105960be1204 (table test-table [id=67b2dbe88ae742518c89bca63736762e]) successfully deleted
I20260925 15:10:13.152657 19812 tablet_replica.cc:330] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: stopping tablet replica
I20260925 15:10:13.153020 19811 ts_tablet_manager.cc:1918] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:13.153007 19812 raft_consensus.cc:2229] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:10:13.153170 19811 log.cc:1196] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/686ea31f724f489ab206105960be1204
I20260925 15:10:13.153797 19811 ts_tablet_manager.cc:1939] T 686ea31f724f489ab206105960be1204 P 576c62eea15646e8b6cdd2e8c8b6f958: Deleting consensus metadata
I20260925 15:10:13.154454 19780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.136970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40634 (local address 127.4.205.193:38727)
0925 15:10:13.137119 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.137126 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.137269 (+ 143us) server_negotiation.cc:407] Connection header received
0925 15:10:13.137462 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.137467 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.137553 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.137687 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:13.139659 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.140628 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.143145 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.143763 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.143885 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.148084 (+ 4199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:13.148095 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:13.148107 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:13.148138 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:13.151103 (+ 2965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.151897 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.151905 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.151910 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.152063 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.152567 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.152572 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.152575 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.153008 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:13.153318 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.153896 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.154124 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:10:13.154927 19812 raft_consensus.cc:2258] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:13.155150 19501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:13.155362 19305 catalog_manager.cc:4841] TS 576c62eea15646e8b6cdd2e8c8b6f958 (127.4.205.194:37485): tablet 686ea31f724f489ab206105960be1204 (table test-table [id=67b2dbe88ae742518c89bca63736762e]) successfully deleted
I20260925 15:10:13.158008 19812 ts_tablet_manager.cc:1905] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:13.167958 19812 ts_tablet_manager.cc:1918] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:13.168164 19812 log.cc:1196] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/686ea31f724f489ab206105960be1204
I20260925 15:10:13.168745 19812 ts_tablet_manager.cc:1939] T 686ea31f724f489ab206105960be1204 P 26f5863e48ea4567a58bfc6f25fb9c1f: Deleting consensus metadata
I20260925 15:10:13.170281 19305 catalog_manager.cc:4841] TS 26f5863e48ea4567a58bfc6f25fb9c1f (127.4.205.195:45485): tablet 686ea31f724f489ab206105960be1204 (table test-table [id=67b2dbe88ae742518c89bca63736762e]) successfully deleted
I20260925 15:10:13.262661 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19358
I20260925 15:10:13.279981 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19510
I20260925 15:10:13.294819 19784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.280761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54424 (local address 127.4.205.194:37485)
0925 15:10:13.280958 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.280964 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.280986 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:10:13.281133 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.281139 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.281227 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.281381 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:13.282728 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.283832 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.285034 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.285776 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.285900 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.288342 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:13.288359 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:13.288377 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:13.288413 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:13.291978 (+ 3565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.292716 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.292726 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.292730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.292874 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.293195 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.293200 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.293203 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.293727 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:13.294040 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.294217 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.294456 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:10:13.295528 19633 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:13.383252 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19510
I20260925 15:10:13.405295 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19641
I20260925 15:10:13.426370 19782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:13.411195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.4.205.195:45485)
0925 15:10:13.411389 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:13.411393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:13.411450 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:10:13.411639 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:13.411643 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:13.411704 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:13.411785 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:13.413657 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.414561 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.416589 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:13.417059 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:13.417169 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:13.420423 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:13.420435 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:13.420447 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:13.420486 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:13.423576 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.424350 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.424359 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.424361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.424475 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:13.424872 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:13.424875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:13.424877 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:13.425230 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:13.425462 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:13.425917 (+ 455us) server_negotiation.cc:299] Negotiation successful
0925 15:10:13.426073 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:10:13.427129 19764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:13.543258 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19641
I20260925 15:10:13.562516 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19288
I20260925 15:10:13.563665 19351 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:13.673095 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19288
I20260925 15:10:13.694582 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--webserver_interface=127.4.205.254
--webserver_port=39781
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:14.005743 19817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:14.005999 19817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:14.006160 19817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:14.013221 19817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:14.013314 19817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:14.013411 19817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:14.013523 19817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:14.013600 19817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:14.022825 19817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=39781
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:14.024026 19817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:14.025285 19817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:14.036710 19822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.036684 19823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.037346 19825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:14.038120 19817 server_base.cc:1034] running on GCE node
I20260925 15:10:14.038838 19817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:14.040194 19817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:14.041396 19817 hybrid_clock.cc:648] HybridClock initialized: now 1790349014041355 us; error 35 us; skew 500 ppm
I20260925 15:10:14.041767 19817 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:14 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349014, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:14.050732 19817 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:14.052186 19817 webserver.cc:460] Webserver started at http://127.4.205.254:39781/ using document root <none> and password file <none>
I20260925 15:10:14.052840 19817 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:14.052944 19817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:14.058259 19817 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:14.060896 19832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.062201 19817 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:10:14.062371 19817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "30d2734b82834aa798d17851baf983ea"
format_stamp: "Formatted at 2026-09-25 15:10:08 on dist-test-slave-64fx"
server_key: "a8f7ef904d41d70953a150c6456cffb6"
server_key_iv: "75fed6adac2b96f47aae9c8735290240"
server_key_version: "encryptionkey@0"
I20260925 15:10:14.063316 19817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:14.084038 19817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:14.088148 19817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:14.088405 19817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:14.106421 19817 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33849
I20260925 15:10:14.106454 19883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33849 every 8 connection(s)
I20260925 15:10:14.107846 19817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:14.114236 19884 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:10:14.114890 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19817
I20260925 15:10:14.115711 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:38727
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=45221
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:14.117445 19884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap starting.
I20260925 15:10:14.136961 19884 log.cc:824] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:14.151672 19884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:10:14.152138 19884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea: Bootstrap complete.
I20260925 15:10:14.160785 19884 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:14.161705 19884 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Initialized, Role: FOLLOWER
I20260925 15:10:14.162277 19884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:14.162519 19884 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:14.162655 19884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:14.162823 19884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:10:14.166483 19884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:14.166936 19884 leader_election.cc:302] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30d2734b82834aa798d17851baf983ea; no voters:
I20260925 15:10:14.167591 19884 leader_election.cc:288] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:10:14.167781 19888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:10:14.168967 19888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [term 3 LEADER]: Becoming Leader. State: Replica: 30d2734b82834aa798d17851baf983ea, State: Running, Role: LEADER
I20260925 15:10:14.169453 19888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } }
I20260925 15:10:14.169714 19884 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:14.172155 19889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:14.172107 19890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30d2734b82834aa798d17851baf983ea. Latest consensus state: current_term: 3 leader_uuid: "30d2734b82834aa798d17851baf983ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30d2734b82834aa798d17851baf983ea" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33849 } } }
I20260925 15:10:14.172540 19889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:14.172571 19890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30d2734b82834aa798d17851baf983ea [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:14.187112 19893 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:14.195276 19893 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:14.196702 19893 catalog_manager.cc:1259] Loaded cluster ID: 1349f125078b48489a9d6545a5a46ab0
I20260925 15:10:14.196789 19893 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:10:14.199015 19893 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f6610d759ad google::LogMessage::Fail() at ??:0
@ 0x7f6610d7888a google::LogMessage::SendToLog() at ??:0
@ 0x7f6610d754de google::LogMessage::Flush() at ??:0
@ 0x7f6610d76a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f66267a8512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f66267a5976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f662682d610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f662682d439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f66139c1345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f66139c491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f66139c4799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f661399feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6616c796db start_thread at ??:0
@ 0x7f660fc0f61f clone at ??:0
W20260925 15:10:14.466027 19886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:14.466323 19886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:14.466488 19886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:14.473311 19886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:14.473460 19886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:14.473672 19886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:10:14.482668 19886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:38727
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=45221
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:14.483843 19886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:14.485086 19886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:14.497704 19926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.497663 19925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.497743 19928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:14.502774 19886 server_base.cc:1034] running on GCE node
I20260925 15:10:14.503314 19886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:14.504148 19886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:14.505364 19886 hybrid_clock.cc:648] HybridClock initialized: now 1790349014505339 us; error 54 us; skew 500 ppm
I20260925 15:10:14.505720 19886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:14 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349014, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:14.508636 19886 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:10:14.510007 19886 webserver.cc:460] Webserver started at http://127.4.205.193:45221/ using document root <none> and password file <none>
I20260925 15:10:14.510617 19886 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:14.510711 19886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:14.515663 19886 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:10:14.518163 19935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.519356 19886 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:10:14.519520 19886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "a2d69e610cb84134a35f845a16d3aa63"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "657a965e007347c96ea417423e296118"
server_key_iv: "96f39515afb0661bac709691fa8c6345"
server_key_version: "encryptionkey@0"
I20260925 15:10:14.520444 19886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:14.531090 19886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:14.535171 19886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:14.535449 19886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:14.536309 19886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:14.537503 19886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:14.537681 19886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.537796 19886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:14.537894 19886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.570871 19886 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:38727
I20260925 15:10:14.570974 20047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:38727 every 8 connection(s)
I20260925 15:10:14.572237 19886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:10:14.576774 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:14.575078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:34791)
0925 15:10:14.575775 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:14.576005 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":263,"threads_started":1}
I20260925 15:10:14.577474 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 19886
W20260925 15:10:14.577521 20048 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:33849 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:33849: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
I20260925 15:10:14.578174 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:37485
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=35255
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:14.578435 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:14.578045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:42733)
0925 15:10:14.578212 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:14.578297 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 15:10:14.579352 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:14.578924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:32827)
0925 15:10:14.579097 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:14.579196 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:10:14.579627 20048 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:10:14.895344 20051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:14.895603 20051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:14.895776 20051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:14.902565 20051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:14.902693 20051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:14.902880 20051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:10:14.911895 20051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:37485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=35255
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:14.913100 20051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:14.914480 20051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:14.927129 20056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.927201 20057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:14.927234 20059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:14.928097 20051 server_base.cc:1034] running on GCE node
I20260925 15:10:14.928655 20051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:14.929534 20051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:14.930795 20051 hybrid_clock.cc:648] HybridClock initialized: now 1790349014930753 us; error 50 us; skew 500 ppm
I20260925 15:10:14.931097 20051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:14 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349014, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:14.934062 20051 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:10:14.935287 20051 webserver.cc:460] Webserver started at http://127.4.205.194:35255/ using document root <none> and password file <none>
I20260925 15:10:14.935840 20051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:14.935928 20051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:14.940378 20051 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:10:14.942936 20066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.944187 20051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 15:10:14.944319 20051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "576c62eea15646e8b6cdd2e8c8b6f958"
format_stamp: "Formatted at 2026-09-25 15:10:09 on dist-test-slave-64fx"
server_key: "536d2e699ddf22136a1c28f226b24436"
server_key_iv: "3f5d1c0191c13e49a119f5f328d730b3"
server_key_version: "encryptionkey@0"
I20260925 15:10:14.945178 20051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:14.963212 20051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:14.967288 20051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:14.967499 20051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:14.968369 20051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:14.969543 20051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:14.969642 20051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:14.969720 20051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:14.969774 20051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:15.001166 20051 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:37485
I20260925 15:10:15.001266 20178 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:37485 every 8 connection(s)
I20260925 15:10:15.002642 20051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:10:15.007139 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.005422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:55053)
0925 15:10:15.006149 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.006362 (+ 213us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":320,"threads_started":1}
W20260925 15:10:15.007758 20179 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:33849 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:33849: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
I20260925 15:10:15.008409 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.008137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:53997)
0925 15:10:15.008263 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.008322 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260925 15:10:15.008906 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20051
I20260925 15:10:15.009250 20181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.008917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:46023)
0925 15:10:15.009091 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.009156 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:10:15.009642 20179 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:10:15.009696 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:45485
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:33849
--webserver_port=34181
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:40741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:15.312201 20182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:15.312484 20182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:15.312637 20182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:15.319435 20182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:15.319527 20182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:15.319720 20182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:10:15.328608 20182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:40741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:45485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=34181
--tserver_master_addrs=127.4.205.254:33849
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:15.329764 20182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:15.330945 20182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:15.342839 20187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:15.342796 20188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:15.343003 20190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:15.343750 20182 server_base.cc:1034] running on GCE node
I20260925 15:10:15.344288 20182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:15.345161 20182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:15.346372 20182 hybrid_clock.cc:648] HybridClock initialized: now 1790349015346337 us; error 45 us; skew 500 ppm
I20260925 15:10:15.346668 20182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:15 dist-test-slave-64fx krb5kdc[18737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349015, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:15.349628 20182 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:10:15.350881 20182 webserver.cc:460] Webserver started at http://127.4.205.195:34181/ using document root <none> and password file <none>
I20260925 15:10:15.351502 20182 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:15.351644 20182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:15.356472 20182 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:10:15.360821 20197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:15.362067 20182 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260925 15:10:15.362232 20182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "26f5863e48ea4567a58bfc6f25fb9c1f"
format_stamp: "Formatted at 2026-09-25 15:10:10 on dist-test-slave-64fx"
server_key: "da076aad7093a817f6409a6f85a6e5af"
server_key_iv: "1dfb9dbec2eb33a3610fffde241484fb"
server_key_version: "encryptionkey@0"
I20260925 15:10:15.363143 20182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:15.405659 20182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:15.409444 20182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:15.409761 20182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:15.410770 20182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:15.412066 20182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:15.412217 20182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:15.412338 20182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:15.412438 20182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:15.444448 20182 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:45485
I20260925 15:10:15.444533 20309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:45485 every 8 connection(s)
I20260925 15:10:15.445995 20182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:10:15.450661 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.448859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:39711)
0925 15:10:15.449607 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.449885 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":279,"threads_started":1}
W20260925 15:10:15.451249 20310 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:33849 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:33849: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
I20260925 15:10:15.451483 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20182
I20260925 15:10:15.452067 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.451725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:55283)
0925 15:10:15.451859 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.451925 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260925 15:10:15.452973 20312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.452666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:38573)
0925 15:10:15.452811 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.452879 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 15:10:15.453310 20310 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:10:15.580946 20314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:15.580219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:51653)
0925 15:10:15.580669 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:15.580738 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":224,"threads_started":1}
I20260925 15:10:16.010771 20315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:16.010116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:58479)
0925 15:10:16.010496 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:16.010580 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":186,"threads_started":1}
I20260925 15:10:16.454516 20316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:16.453842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:46859)
0925 15:10:16.454240 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:16.454312 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":200,"threads_started":1}
I20260925 15:10:16.582499 20317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:16.581853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:44601)
0925 15:10:16.582274 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:16.582350 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":217,"threads_started":1}
I20260925 15:10:17.012475 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:17.011811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:52039)
0925 15:10:17.012261 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:17.012334 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":243,"threads_started":1}
I20260925 15:10:17.455857 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:17.455246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:41053)
0925 15:10:17.455635 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:17.455706 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":203,"threads_started":1}
I20260925 15:10:17.583884 20320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:17.583224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:47577)
0925 15:10:17.583633 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:17.583712 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":223,"threads_started":1}
I20260925 15:10:18.014034 20321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:18.013337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:47989)
0925 15:10:18.013795 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:18.013882 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":292,"threads_started":1}
I20260925 15:10:18.457434 20322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:18.456785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:40823)
0925 15:10:18.457202 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:18.457276 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":228,"threads_started":1}
I20260925 15:10:18.585386 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:18.584745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:57271)
0925 15:10:18.585139 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:18.585211 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":208,"threads_started":1}
I20260925 15:10:19.015594 20324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:19.014908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:34109)
0925 15:10:19.015320 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:19.015397 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":315,"threads_started":1}
I20260925 15:10:19.458843 20325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:19.458194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:45043)
0925 15:10:19.458626 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:19.458698 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":249,"threads_started":1}
I20260925 15:10:19.586825 20326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:19.586193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:35865)
0925 15:10:19.586594 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:19.586669 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":215,"threads_started":1}
I20260925 15:10:20.018493 20327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:20.017823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:35123)
0925 15:10:20.018249 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:20.018322 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":342,"thread_start_us":226,"threads_started":1}
I20260925 15:10:20.460414 20328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:20.459634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:46749)
0925 15:10:20.460155 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:20.460234 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":310,"threads_started":1}
I20260925 15:10:20.588415 20329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:20.587684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:42319)
0925 15:10:20.588105 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:20.588190 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":221,"threads_started":1}
I20260925 15:10:21.019841 20330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:21.019231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:55801)
0925 15:10:21.019615 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:21.019696 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":198,"threads_started":1}
I20260925 15:10:21.461941 20331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:21.461287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:54531)
0925 15:10:21.461699 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:21.461779 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":205,"threads_started":1}
I20260925 15:10:21.590011 20332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:21.589274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:37321)
0925 15:10:21.589739 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:21.589813 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":190,"threads_started":1}
I20260925 15:10:22.021683 20333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:22.020790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:33041)
0925 15:10:22.021247 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:22.021372 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":207,"threads_started":1}
I20260925 15:10:22.463599 20334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:22.462865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:50295)
0925 15:10:22.463308 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:22.463393 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":206,"threads_started":1}
I20260925 15:10:22.591375 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:22.590771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:55349)
0925 15:10:22.591159 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:22.591228 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":205,"threads_started":1}
I20260925 15:10:23.023134 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:23.022493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:33161)
0925 15:10:23.022893 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:23.022965 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260925 15:10:23.465194 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:23.464450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:48195)
0925 15:10:23.464919 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:23.465017 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":252,"threads_started":1}
I20260925 15:10:23.592796 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:23.592135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:47609)
0925 15:10:23.592561 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:23.592639 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":210,"threads_started":1}
I20260925 15:10:24.024771 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:24.024005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:39573)
0925 15:10:24.024495 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:24.024565 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":249,"threads_started":1}
I20260925 15:10:24.466800 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:24.466085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:45535)
0925 15:10:24.466525 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:24.466617 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":243,"threads_started":1}
I20260925 15:10:24.594312 20341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:24.593586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:34531)
0925 15:10:24.594089 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:24.594165 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":275,"threads_started":1}
I20260925 15:10:25.026479 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:25.025692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:53991)
0925 15:10:25.026184 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:25.026260 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":292,"threads_started":1}
I20260925 15:10:25.468230 20343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:25.467592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:41267)
0925 15:10:25.468002 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:25.468077 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":219,"threads_started":1}
I20260925 15:10:25.595945 20344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:25.595238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:41567)
0925 15:10:25.595640 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:25.595717 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":199,"threads_started":1}
I20260925 15:10:26.027987 20345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:26.027323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:34799)
0925 15:10:26.027728 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:26.027815 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260925 15:10:26.469702 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:26.468982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:53583)
0925 15:10:26.469369 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:26.469450 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":193,"threads_started":1}
I20260925 15:10:26.597482 20347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:26.596820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:54827)
0925 15:10:26.597258 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:26.597335 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":229,"threads_started":1}
I20260925 15:10:27.029539 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:27.028924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:41033)
0925 15:10:27.029311 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:27.029386 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":203,"threads_started":1}
I20260925 15:10:27.471096 20349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:27.470491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:44667)
0925 15:10:27.470883 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:27.470957 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":205,"threads_started":1}
I20260925 15:10:27.599140 20350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:27.598333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:55295)
0925 15:10:27.598840 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:27.598957 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":263,"threads_started":1}
I20260925 15:10:28.031083 20351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:28.030358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:58021)
0925 15:10:28.030786 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:28.030873 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":208,"threads_started":1}
I20260925 15:10:28.472698 20352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:28.472004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:49205)
0925 15:10:28.472439 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:28.472524 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":235,"threads_started":1}
I20260925 15:10:28.600770 20353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:28.600084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:59723)
0925 15:10:28.600514 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:28.600599 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":224,"threads_started":1}
I20260925 15:10:29.032658 20354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:29.031984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:55521)
0925 15:10:29.032390 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:29.032483 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260925 15:10:29.474385 20355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:29.473740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:57643)
0925 15:10:29.474137 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:29.474213 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":210,"threads_started":1}
I20260925 15:10:29.602631 20356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:29.601942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.193:48233)
0925 15:10:29.602354 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:29.602468 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":213,"threads_started":1}
I20260925 15:10:30.034512 20357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:30.033848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.194:56073)
0925 15:10:30.034253 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:30.034336 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":211,"threads_started":1}
I20260925 15:10:30.455602 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 19886
I20260925 15:10:30.472582 20358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:30.456622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.4.205.193:38727)
0925 15:10:30.457213 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:30.457232 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:30.457351 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:10:30.459611 (+ 2260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:30.459620 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:30.460035 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:30.460325 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:30.461625 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.462829 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.463969 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.464614 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.464764 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:30.467227 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:30.467234 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:30.467246 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:30.467283 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:30.469916 (+ 2633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.470532 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.470538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.470544 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.470727 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.471030 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.471034 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.471036 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.471485 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:30.471709 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:30.471998 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:10:30.472282 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":298,"threads_started":1}
I20260925 15:10:30.473915 20043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:30.476411 20360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:30.475546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33849 (local address 127.4.205.195:41145)
0925 15:10:30.476051 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:30.476190 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:33849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":248,"threads_started":1}
I20260925 15:10:30.556864 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 19886
I20260925 15:10:30.573267 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20051
I20260925 15:10:30.590715 20361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:30.574188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52822 (local address 127.4.205.194:37485)
0925 15:10:30.574715 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:30.574728 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:30.574800 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:10:30.576696 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:30.576705 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:30.577122 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:30.577453 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:30.578963 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.580144 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.581354 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.582098 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.582285 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:30.584723 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:30.584731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:30.584746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:30.584782 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:30.587564 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.588204 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.588211 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.588217 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.588392 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.588756 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.588762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.588768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.589255 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:30.589627 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:30.590060 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 15:10:30.590445 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":240,"threads_started":1}
I20260925 15:10:30.591986 20174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:30.674546 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20051
I20260925 15:10:30.692263 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20182
I20260925 15:10:30.709439 20363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:30.693317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.4.205.195:45485)
0925 15:10:30.693985 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:30.694003 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:30.694042 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:10:30.695613 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:30.695622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:30.696031 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:30.696303 (+ 272us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:30.698023 (+ 1720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.699202 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.700428 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:30.701029 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:30.701151 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:30.703868 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:30.703876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:30.703890 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:30.703938 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:30.706775 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.707387 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.707393 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.707405 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.707552 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:30.707832 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:30.707835 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:30.707837 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:30.708323 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:30.708562 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:30.708861 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 15:10:30.709161 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":345,"threads_started":1}
I20260925 15:10:30.710737 20305 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:30.791846 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20182
2026-09-25T15:10:30Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (23514 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:10:30 dist-test-slave-64fx krb5kdc[20367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:10:30 dist-test-slave-64fx krb5kdc[20367](info): set up 2 sockets
Sep 25 15:10:30 dist-test-slave-64fx krb5kdc[20367](info): commencing operation
krb5kdc: starting...
W20260925 15:10:31.829946 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.000s user 0.007s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:10:31 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:10:32.030241 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:32.340588 20380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:32.340821 20380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:32.340898 20380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:32.347718 20380 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:10:32.347795 20380 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:10:32.347847 20380 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:10:32.347888 20380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:32.347959 20380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:32.348081 20380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:32.348145 20380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:32.348246 20380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:32.357659 20380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:32.358770 20380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:32.359946 20380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:32.370970 20388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:32.371197 20385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:32.371089 20386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:32.372112 20380 server_base.cc:1034] running on GCE node
I20260925 15:10:32.372790 20380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:32.374382 20380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:32.375602 20380 hybrid_clock.cc:648] HybridClock initialized: now 1790349032375555 us; error 26 us; skew 500 ppm
I20260925 15:10:32.375910 20380 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:32 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:32.378947 20380 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:32.380285 20380 webserver.cc:460] Webserver started at http://127.4.205.254:33593/ using document root <none> and password file <none>
I20260925 15:10:32.380923 20380 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:32.381044 20380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:32.381372 20380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:32.384660 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "6bd8362500b94bc3bc8be6bab840cc16"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "d3e402d0226308cf60d9d79110d137f6"
server_key_iv: "915ca3acad01d4f7b5501c57235eb17c"
server_key_version: "encryptionkey@0"
I20260925 15:10:32.385406 20380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "6bd8362500b94bc3bc8be6bab840cc16"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "d3e402d0226308cf60d9d79110d137f6"
server_key_iv: "915ca3acad01d4f7b5501c57235eb17c"
server_key_version: "encryptionkey@0"
I20260925 15:10:32.389968 20380 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:10:32.393057 20395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:32.394441 20380 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:32.394622 20380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "6bd8362500b94bc3bc8be6bab840cc16"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "d3e402d0226308cf60d9d79110d137f6"
server_key_iv: "915ca3acad01d4f7b5501c57235eb17c"
server_key_version: "encryptionkey@0"
I20260925 15:10:32.394951 20380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:32.424070 20380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:32.428167 20380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:32.428367 20380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:32.445538 20380 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44489
I20260925 15:10:32.445546 20446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44489 every 8 connection(s)
I20260925 15:10:32.446768 20380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:32.450126 20447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:32.450809 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20380
I20260925 15:10:32.451051 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:10:32.451822 4919 external_mini_cluster.cc:1428] Setting key f9ce28fa084922e54af3fdbb3afb1ddc
I20260925 15:10:32.463537 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap starting.
Sep 25 15:10:32 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:32.467096 20447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:32.468406 20447 log.cc:824] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:32.471351 20447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: No bootstrap required, opened a new log
I20260925 15:10:32.474920 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:32.453830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.4.205.254:44489)
0925 15:10:32.454569 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:32.454582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:32.454623 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:10:32.456249 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:32.456269 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:32.456601 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:32.456899 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:32.458579 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:32.459721 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:32.461647 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:32.462247 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:32.462371 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:32.466975 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:32.466991 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:32.467011 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:32.467052 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:32.470663 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:32.471319 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:32.471329 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:32.471343 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:32.471550 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:32.471934 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:32.471939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:32.471942 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:32.472556 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:32.472858 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:32.473164 (+ 306us) server_negotiation.cc:299] Negotiation successful
0925 15:10:32.473608 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":348,"threads_started":1}
I20260925 15:10:32.480000 20447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:32.480429 20447 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:32.480543 20447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Initialized, Role: FOLLOWER
I20260925 15:10:32.481057 20447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:32.481312 20447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:32.481477 20447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:32.481678 20447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:32.484019 20447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:32.484544 20447 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd8362500b94bc3bc8be6bab840cc16; no voters:
I20260925 15:10:32.485061 20447 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:10:32.485224 20452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:32.485850 20452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 LEADER]: Becoming Leader. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Running, Role: LEADER
I20260925 15:10:32.486395 20452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:32.486860 20447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:32.488757 20454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd8362500b94bc3bc8be6bab840cc16. Latest consensus state: current_term: 1 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:10:32.489102 20454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:32.489578 20453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:10:32.489905 20453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:32.491895 20460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:32.509737 20460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:32.518721 20460 catalog_manager.cc:1347] Generated new cluster ID: 2883f45bce434fecad4ed92e9759f84c
I20260925 15:10:32.518831 20460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:32.537070 20460 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:10:32.538733 20460 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:32.547899 20460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Generated new TSK 0
I20260925 15:10:32.548806 20460 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:10:32.614873 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:32.947988 20473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:32.948230 20473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:32.948377 20473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:32.954869 20473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:32.954958 20473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:32.955119 20473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:10:32.964197 20473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:32.965340 20473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:32.966678 20473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:32.979262 20481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:32.979324 20478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:32.979352 20479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:32.980048 20473 server_base.cc:1034] running on GCE node
I20260925 15:10:32.980625 20473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:32.981477 20473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:32.982661 20473 hybrid_clock.cc:648] HybridClock initialized: now 1790349032982613 us; error 43 us; skew 500 ppm
I20260925 15:10:32.982939 20473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:32 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:32.985980 20473 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:10:32.987469 20473 webserver.cc:460] Webserver started at http://127.4.205.193:44117/ using document root <none> and password file <none>
I20260925 15:10:32.988157 20473 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:32.988282 20473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:32.988600 20473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:32.991748 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "b9d1a4c26b3949af8ababf853ee67cad"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "41ae06e97a76ac5c8e5d9aa321bd3d26"
server_key_iv: "31df09106cb252f1c7b28e6ed4553c53"
server_key_version: "encryptionkey@0"
I20260925 15:10:32.992499 20473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "b9d1a4c26b3949af8ababf853ee67cad"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "41ae06e97a76ac5c8e5d9aa321bd3d26"
server_key_iv: "31df09106cb252f1c7b28e6ed4553c53"
server_key_version: "encryptionkey@0"
I20260925 15:10:32.997185 20473 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.005s
I20260925 15:10:33.000351 20488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.001956 20473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:33.002126 20473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "b9d1a4c26b3949af8ababf853ee67cad"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "41ae06e97a76ac5c8e5d9aa321bd3d26"
server_key_iv: "31df09106cb252f1c7b28e6ed4553c53"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.002321 20473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:33.036196 20473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:33.039893 20473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:33.040184 20473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:33.041045 20473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:33.042297 20473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:33.042407 20473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.042572 20473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:33.042641 20473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.073998 20473 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:37887
I20260925 15:10:33.074090 20600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:37887 every 8 connection(s)
I20260925 15:10:33.075165 20473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:10:33.078913 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20473
I20260925 15:10:33.079114 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:10:33.079726 4919 external_mini_cluster.cc:1428] Setting key 6b842cc3505c8676a477b0890b97170c
Sep 25 15:10:33 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349032, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:33.094549 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:33.077276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:55389 (local address 127.4.205.254:44489)
0925 15:10:33.077757 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:33.077762 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:33.078539 (+ 777us) server_negotiation.cc:407] Connection header received
0925 15:10:33.080408 (+ 1869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:33.080414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:33.080500 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:33.080624 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:33.082082 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.082765 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.084226 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.084634 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.084772 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:33.088527 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:33.088536 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:33.088541 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:33.088580 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:33.091740 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:33.092386 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:33.092394 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:33.092402 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:33.092512 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:33.092909 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:33.092915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:33.092920 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:33.093142 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:33.093296 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:33.093961 (+ 665us) server_negotiation.cc:299] Negotiation successful
0925 15:10:33.094193 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":273,"threads_started":1}
I20260925 15:10:33.095352 20604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:33.077765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:55389)
0925 15:10:33.078398 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:33.078433 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:33.080145 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:33.080839 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:33.080847 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:33.081342 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:33.081923 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:33.081940 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.082915 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.082919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:33.084079 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:33.084091 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.084856 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.084866 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:33.085094 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:33.085760 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:33.085785 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:33.088327 (+ 2542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:33.091869 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:33.091878 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:33.091892 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:33.092265 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:33.092625 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:33.092631 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:33.092635 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:33.092802 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:33.093256 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:33.093262 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:33.093553 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:10:33.093872 (+ 319us) client_negotiation.cc:240] Negotiation successful
0925 15:10:33.094326 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":250,"threads_started":1}
I20260925 15:10:33.096763 20601 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:33.097060 20601 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:33.097800 20601 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:33.100193 20412 ts_manager.cc:194] Registered new tserver with Master: b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887)
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:10:33.102468 20412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:55389
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:10:33.129189 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:33.430074 20607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:33.430308 20607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:33.430512 20607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:33.436880 20607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:33.436983 20607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:33.437155 20607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:10:33.446044 20607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:33.447098 20607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:33.448273 20607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:33.460805 20612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:33.460811 20615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:33.461442 20613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:33.461490 20607 server_base.cc:1034] running on GCE node
I20260925 15:10:33.462050 20607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:33.462896 20607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:33.464129 20607 hybrid_clock.cc:648] HybridClock initialized: now 1790349033464082 us; error 39 us; skew 500 ppm
I20260925 15:10:33.464440 20607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:33 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:33.467281 20607 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:10:33.468545 20607 webserver.cc:460] Webserver started at http://127.4.205.194:41565/ using document root <none> and password file <none>
I20260925 15:10:33.469148 20607 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:33.469295 20607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:33.469647 20607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:33.472215 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "7857a4cf00eb466dab5f1debf7479d97"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "f9a705ba1b5f352dd29804d11166eefa"
server_key_iv: "8a1db3a1d7de109db38d39185e809326"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.473060 20607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "7857a4cf00eb466dab5f1debf7479d97"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "f9a705ba1b5f352dd29804d11166eefa"
server_key_iv: "8a1db3a1d7de109db38d39185e809326"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.477820 20607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 15:10:33.480954 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.482168 20607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:33.482354 20607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "7857a4cf00eb466dab5f1debf7479d97"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "f9a705ba1b5f352dd29804d11166eefa"
server_key_iv: "8a1db3a1d7de109db38d39185e809326"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.482561 20607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:33.500445 20607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:33.503944 20607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:33.504235 20607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:33.505108 20607 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:33.506382 20607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:33.506474 20607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.506608 20607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:33.506676 20607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.538220 20607 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:46881
I20260925 15:10:33.538307 20734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:46881 every 8 connection(s)
I20260925 15:10:33.539479 20607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:10:33.547120 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20607
I20260925 15:10:33.547310 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:10:33.547748 4919 external_mini_cluster.cc:1428] Setting key d38d2f9031751f07f8b22efb3b4cc4d0
Sep 25 15:10:33 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:33.559027 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:33.542163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:35047 (local address 127.4.205.254:44489)
0925 15:10:33.542362 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:33.542366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:33.543389 (+ 1023us) server_negotiation.cc:407] Connection header received
0925 15:10:33.545157 (+ 1768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:33.545163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:33.545257 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:33.545394 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:33.546858 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.547706 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.549653 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.550056 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.550185 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:33.553626 (+ 3441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:33.553631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:33.553634 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:33.553665 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:33.556098 (+ 2433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:33.556813 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:33.556825 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:33.556828 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:33.556951 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:33.557348 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:33.557354 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:33.557359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:33.557691 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:33.557842 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:33.558469 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 15:10:33.558691 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:10:33.560058 20737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:33.542538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:35047)
0925 15:10:33.543236 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:33.543270 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:33.544893 (+ 1623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:33.545677 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:33.545685 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:33.546178 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:33.546693 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:33.546709 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.547851 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.547861 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:33.549490 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:33.549501 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:33.550275 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:33.550281 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:33.550502 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:33.551140 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:33.551164 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:33.553401 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:33.556235 (+ 2834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:33.556244 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:33.556259 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:33.556690 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:33.557056 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:33.557063 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:33.557067 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:33.557238 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:33.557813 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:33.557820 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:33.558134 (+ 314us) client_negotiation.cc:769] Sending connection context
0925 15:10:33.558385 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 15:10:33.558860 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":277,"threads_started":1}
I20260925 15:10:33.561640 20735 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:33.561932 20735 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:33.562623 20735 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:33.564087 20412 ts_manager.cc:194] Registered new tserver with Master: 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881)
I20260925 15:10:33.565307 20412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:35047
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:10:33.598273 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:33.902613 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:33.902843 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:33.902926 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:33.909852 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:33.909941 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:33.910039 20740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:10:33.918884 20740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:33.919979 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:33.921166 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:33.933597 20745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:33.933558 20746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:33.934162 20748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:33.934646 20740 server_base.cc:1034] running on GCE node
I20260925 15:10:33.935105 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:33.935932 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:33.937144 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790349033937090 us; error 51 us; skew 500 ppm
I20260925 15:10:33.937436 20740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:33 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:33.940307 20740 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:10:33.941708 20740 webserver.cc:460] Webserver started at http://127.4.205.195:41205/ using document root <none> and password file <none>
I20260925 15:10:33.942322 20740 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:33.942464 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:33.942786 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:10:33.945533 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "2102353d0f3746fe9908cdbd15be6619"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "5f1588ff6248fcdfc39cdaa7e8e70c22"
server_key_iv: "aed2a890b7f13aa3282d104f2669528f"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.946336 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "2102353d0f3746fe9908cdbd15be6619"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "5f1588ff6248fcdfc39cdaa7e8e70c22"
server_key_iv: "aed2a890b7f13aa3282d104f2669528f"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.950922 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:10:33.954001 20755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.955173 20740 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:10:33.955363 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "2102353d0f3746fe9908cdbd15be6619"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "5f1588ff6248fcdfc39cdaa7e8e70c22"
server_key_iv: "aed2a890b7f13aa3282d104f2669528f"
server_key_version: "encryptionkey@0"
I20260925 15:10:33.955538 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:33.977748 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:33.981698 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:33.981959 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:33.982838 20740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:33.983911 20740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:33.984004 20740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:33.984162 20740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:33.984246 20740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:34.015664 20740 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34053
I20260925 15:10:34.015743 20867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34053 every 8 connection(s)
I20260925 15:10:34.016876 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:10:34.018332 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20740
I20260925 15:10:34.018517 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:10:34.019099 4919 external_mini_cluster.cc:1428] Setting key 753fa2d54862d6f5e9b6f08dc2cd2608
Sep 25 15:10:34 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349033, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:34.036821 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.019397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:55793 (local address 127.4.205.254:44489)
0925 15:10:34.019586 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.019591 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.021704 (+ 2113us) server_negotiation.cc:407] Connection header received
0925 15:10:34.023608 (+ 1904us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.023613 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.023692 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.023792 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:34.025217 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.026062 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.027570 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.027963 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.028089 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.031360 (+ 3271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:34.031365 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:34.031368 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:34.031400 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:34.034401 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.035005 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.035009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.035011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.035133 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.035494 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.035496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.035498 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.035732 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:34.035859 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.036386 (+ 527us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.036527 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:10:34.037798 20870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.020043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:55793)
0925 15:10:34.021024 (+ 981us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.021066 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.023338 (+ 2272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.024030 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.024038 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.024539 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:34.025056 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.025074 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.026213 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.026219 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.027423 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.027435 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.028183 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.028189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.028426 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.029023 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:34.029042 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:34.031169 (+ 2127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:34.034509 (+ 3340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:34.034515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:34.034525 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:34.034907 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:34.035220 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:34.035224 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:34.035226 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:34.035405 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:34.035840 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:34.035845 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:34.036113 (+ 268us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.036319 (+ 206us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.036647 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":730,"thread_start_us":522,"threads_started":1}
I20260925 15:10:34.038960 20868 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:34.039222 20868 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:34.039831 20868 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:34.041158 20412 ts_manager.cc:194] Registered new tserver with Master: 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053)
I20260925 15:10:34.042011 20412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:55793
I20260925 15:10:34.046036 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:10:34.066419 20603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.050098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52438 (local address 127.4.205.254:44489)
0925 15:10:34.050404 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.050409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.050698 (+ 289us) server_negotiation.cc:407] Connection header received
0925 15:10:34.050807 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.050810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.050859 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.050927 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:34.052183 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.052782 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.054029 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.054437 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.054525 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.061815 (+ 7290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:34.061819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:34.061822 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:34.061849 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:34.064225 (+ 2376us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.064813 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.064817 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.064819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.064924 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.065214 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.065217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.065219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.065515 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:34.065736 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.065968 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.066100 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260925 15:10:34.069881 20412 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52438:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:10:34.076274 20412 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:10:34.105414 20601 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
I20260925 15:10:34.110284 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.087930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60100 (local address 127.4.205.194:46881)
0925 15:10:34.101589 (+ 13659us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.101599 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.101628 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:10:34.101729 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.101742 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.102063 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.102277 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.103176 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.104765 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.107426 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.109174 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.109320 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.109524 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.109808 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.109969 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13468,"thread_start_us":13551,"threads_started":1}
I20260925 15:10:34.110351 20880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.087868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46881 (local address 127.0.0.1:60100)
0925 15:10:34.088602 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.088641 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.088805 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.102327 (+ 13522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.102338 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.102384 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.102879 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.102905 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.104938 (+ 2033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.104942 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.107279 (+ 2337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.107291 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.109369 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.109375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.109592 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.109641 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.109769 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.109949 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":24,"thread_start_us":360,"threads_started":1}
I20260925 15:10:34.111985 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.088937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48084 (local address 127.4.205.193:37887)
0925 15:10:34.102348 (+ 13411us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.102358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.102384 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:10:34.102483 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.102491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.102653 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.102858 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.103632 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.105112 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.108998 (+ 3886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.110452 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.110574 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.110786 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.111476 (+ 690us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.111645 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13240,"thread_start_us":13076,"threads_started":1}
I20260925 15:10:34.112051 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.089476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39698 (local address 127.4.205.195:34053)
0925 15:10:34.100290 (+ 10814us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.100303 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.103630 (+ 3327us) server_negotiation.cc:407] Connection header received
0925 15:10:34.103763 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.103771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.103976 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.104149 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.105072 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.106699 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.109132 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.110452 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.110574 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.110798 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.111505 (+ 707us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.111666 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10598,"thread_start_us":10832,"threads_started":1}
I20260925 15:10:34.112687 20882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.089127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34053 (local address 127.0.0.1:39698)
0925 15:10:34.103504 (+ 14377us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.103523 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.103630 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.104431 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.104436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.104479 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.104844 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.104857 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.106949 (+ 2092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.106956 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.108984 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.108999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.110967 (+ 1968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.110971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.111165 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.111310 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.111395 (+ 85us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.112307 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14228,"mutex_wait_us":891,"thread_start_us":11501,"threads_started":1}
I20260925 15:10:34.112836 20884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.088362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:37887 (local address 127.0.0.1:48084)
0925 15:10:34.100975 (+ 12613us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.101000 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.101139 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.102844 (+ 1705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.102849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.102874 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.103406 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.103419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.105315 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.105320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.108850 (+ 3530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.108861 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.110954 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.110961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.111163 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.111309 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.111395 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.112475 (+ 1080us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12435,"mutex_wait_us":88,"thread_start_us":14831,"threads_started":1}
I20260925 15:10:34.114363 20670 tablet_service.cc:1467] Processing CreateTablet for tablet 07df0513fb704de2b84239826c73e33e (DEFAULT_TABLE table=test-table [id=53e39f83ebd34d46aea75ca294781236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:34.115532 20670 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df0513fb704de2b84239826c73e33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:34.116144 20536 tablet_service.cc:1467] Processing CreateTablet for tablet 07df0513fb704de2b84239826c73e33e (DEFAULT_TABLE table=test-table [id=53e39f83ebd34d46aea75ca294781236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:34.116144 20803 tablet_service.cc:1467] Processing CreateTablet for tablet 07df0513fb704de2b84239826c73e33e (DEFAULT_TABLE table=test-table [id=53e39f83ebd34d46aea75ca294781236]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:34.117493 20536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df0513fb704de2b84239826c73e33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:34.117497 20803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07df0513fb704de2b84239826c73e33e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:34.125855 20886 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Bootstrap starting.
I20260925 15:10:34.129483 20886 tablet_bootstrap.cc:654] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:34.130658 20886 log.cc:824] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:34.131096 20887 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Bootstrap starting.
I20260925 15:10:34.134189 20888 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Bootstrap starting.
I20260925 15:10:34.135673 20887 tablet_bootstrap.cc:654] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:34.136960 20888 tablet_bootstrap.cc:654] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:34.137194 20887 log.cc:824] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:34.138077 20888 log.cc:824] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:34.139673 20886 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: No bootstrap required, opened a new log
I20260925 15:10:34.139911 20887 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: No bootstrap required, opened a new log
I20260925 15:10:34.139923 20886 ts_tablet_manager.cc:1397] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260925 15:10:34.140143 20887 ts_tablet_manager.cc:1397] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:10:34.140232 20888 tablet_bootstrap.cc:492] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: No bootstrap required, opened a new log
I20260925 15:10:34.140441 20888 ts_tablet_manager.cc:1397] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260925 15:10:34.147425 20888 raft_consensus.cc:348] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.147794 20888 raft_consensus.cc:374] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:34.147905 20888 raft_consensus.cc:729] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d1a4c26b3949af8ababf853ee67cad, State: Initialized, Role: FOLLOWER
I20260925 15:10:34.148478 20888 consensus_queue.cc:260] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.150065 20888 ts_tablet_manager.cc:1428] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20260925 15:10:34.150198 20886 raft_consensus.cc:348] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.150198 20887 raft_consensus.cc:348] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.150588 20887 raft_consensus.cc:374] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:34.150645 20886 raft_consensus.cc:374] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:34.150697 20887 raft_consensus.cc:729] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2102353d0f3746fe9908cdbd15be6619, State: Initialized, Role: FOLLOWER
I20260925 15:10:34.150764 20886 raft_consensus.cc:729] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7857a4cf00eb466dab5f1debf7479d97, State: Initialized, Role: FOLLOWER
I20260925 15:10:34.151253 20887 consensus_queue.cc:260] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.151296 20886 consensus_queue.cc:260] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.152640 20868 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
I20260925 15:10:34.152812 20886 ts_tablet_manager.cc:1428] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:10:34.153159 20887 ts_tablet_manager.cc:1428] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 15:10:34.153697 20735 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
W20260925 15:10:34.268491 20869 tablet.cc:2369] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:10:34.291460 20736 tablet.cc:2369] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:10:34.327503 20602 tablet.cc:2369] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:10:34.481632 20894 raft_consensus.cc:482] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:34.481937 20894 raft_consensus.cc:504] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.483546 20894 leader_election.cc:288] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053), b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887)
I20260925 15:10:34.484679 20892 raft_consensus.cc:482] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:34.485066 20892 raft_consensus.cc:504] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.487138 20892 leader_election.cc:288] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881), 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053)
I20260925 15:10:34.492260 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.483869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:36519 (local address 127.4.205.195:34053)
0925 15:10:34.484100 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.484108 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.484823 (+ 715us) server_negotiation.cc:407] Connection header received
0925 15:10:34.485050 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.485054 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.485112 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.485208 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.485863 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.486879 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.489731 (+ 2852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.491008 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.491140 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.491212 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.491817 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.491982 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:10:34.492336 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.484046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:42415 (local address 127.4.205.193:37887)
0925 15:10:34.484313 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.484318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.484993 (+ 675us) server_negotiation.cc:407] Connection header received
0925 15:10:34.485281 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.485289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.485385 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.485460 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.486285 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.487384 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.489940 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.491176 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.491301 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.491377 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.491916 (+ 539us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.492082 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":68}
I20260925 15:10:34.493080 20895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.483910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:37887 (local address 127.4.205.194:42415)
0925 15:10:34.484893 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.484912 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.485107 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.485702 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.485708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.485734 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.486119 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.486133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.487575 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.487585 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.489779 (+ 2194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.489803 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.491544 (+ 1741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.491549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.491704 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.491759 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.492509 (+ 750us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.492662 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":861,"mutex_wait_us":28,"thread_start_us":370,"threads_started":1}
I20260925 15:10:34.493166 20896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.483701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34053 (local address 127.4.205.194:36519)
0925 15:10:34.484686 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.484707 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.484900 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.485221 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.485226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.485250 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.485706 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.485716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.487029 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.487034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.489502 (+ 2468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.489515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.491433 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.491440 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.491620 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.491652 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.492697 (+ 1045us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.492817 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":65,"thread_start_us":860,"threads_started":1}
I20260925 15:10:34.494329 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "7857a4cf00eb466dab5f1debf7479d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d1a4c26b3949af8ababf853ee67cad" is_pre_election: true
I20260925 15:10:34.494455 20823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "7857a4cf00eb466dab5f1debf7479d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2102353d0f3746fe9908cdbd15be6619" is_pre_election: true
I20260925 15:10:34.494753 20556 raft_consensus.cc:2454] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7857a4cf00eb466dab5f1debf7479d97 in term 0.
I20260925 15:10:34.494897 20823 raft_consensus.cc:2454] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7857a4cf00eb466dab5f1debf7479d97 in term 0.
I20260925 15:10:34.495523 20623 leader_election.cc:302] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7857a4cf00eb466dab5f1debf7479d97, b9d1a4c26b3949af8ababf853ee67cad; no voters:
I20260925 15:10:34.495945 20894 raft_consensus.cc:2781] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:10:34.496110 20894 raft_consensus.cc:482] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:10:34.496225 20894 raft_consensus.cc:3037] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:34.496572 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.487760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:59259 (local address 127.4.205.194:46881)
0925 15:10:34.487964 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.487971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.488007 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:10:34.488057 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.488061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.488123 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.488216 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.488981 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.490445 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.494136 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.495376 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.495498 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.495573 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.496209 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.496333 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":62}
I20260925 15:10:34.496769 20897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.487220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46881 (local address 127.4.205.193:59259)
0925 15:10:34.487705 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.487722 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.487831 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.488388 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.488406 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.488436 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.488806 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.488822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.490665 (+ 1843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.490672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.493948 (+ 3276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.493965 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.495777 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.495783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.495970 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.496002 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.496163 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.496300 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"mutex_wait_us":26,"thread_start_us":726,"threads_started":1}
I20260925 15:10:34.498600 20894 raft_consensus.cc:504] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.499083 20894 leader_election.cc:288] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [CANDIDATE]: Term 1 election: Requested vote from peers 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053), b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887)
I20260925 15:10:34.499531 20556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "7857a4cf00eb466dab5f1debf7479d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b9d1a4c26b3949af8ababf853ee67cad"
I20260925 15:10:34.499703 20556 raft_consensus.cc:3037] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:34.499661 20823 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "7857a4cf00eb466dab5f1debf7479d97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2102353d0f3746fe9908cdbd15be6619"
I20260925 15:10:34.499886 20823 raft_consensus.cc:3037] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:34.500172 20690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7857a4cf00eb466dab5f1debf7479d97" is_pre_election: true
I20260925 15:10:34.500643 20690 raft_consensus.cc:2379] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b9d1a4c26b3949af8ababf853ee67cad in current term 1: Already voted for candidate 7857a4cf00eb466dab5f1debf7479d97 in this term.
I20260925 15:10:34.501117 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.488686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:42819 (local address 127.4.205.195:34053)
0925 15:10:34.492504 (+ 3818us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.492509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.494109 (+ 1600us) server_negotiation.cc:407] Connection header received
0925 15:10:34.494987 (+ 878us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.494992 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.495080 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.495136 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:34.495776 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.496780 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.498658 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.499762 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.499953 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.500034 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.500693 (+ 659us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.500825 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3718,"thread_start_us":4603,"threads_started":1}
I20260925 15:10:34.501895 20898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.488281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34053 (local address 127.4.205.193:42819)
0925 15:10:34.493937 (+ 5656us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:34.493954 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:34.494818 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:34.495301 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:34.495306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:34.495328 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:34.495639 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.495648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.496901 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.496905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.498520 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:34.498531 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.500181 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.500188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:34.500501 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.500535 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:10:34.501303 (+ 768us) client_negotiation.cc:240] Negotiation successful
0925 15:10:34.501437 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5540,"mutex_wait_us":3,"thread_start_us":5327,"threads_started":1}
I20260925 15:10:34.502763 20822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07df0513fb704de2b84239826c73e33e" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2102353d0f3746fe9908cdbd15be6619" is_pre_election: true
I20260925 15:10:34.503017 20556 raft_consensus.cc:2454] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7857a4cf00eb466dab5f1debf7479d97 in term 1.
I20260925 15:10:34.503191 20823 raft_consensus.cc:2454] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7857a4cf00eb466dab5f1debf7479d97 in term 1.
I20260925 15:10:34.503492 20623 leader_election.cc:302] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7857a4cf00eb466dab5f1debf7479d97, b9d1a4c26b3949af8ababf853ee67cad; no voters:
I20260925 15:10:34.503613 20492 leader_election.cc:302] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b9d1a4c26b3949af8ababf853ee67cad; no voters: 2102353d0f3746fe9908cdbd15be6619, 7857a4cf00eb466dab5f1debf7479d97
I20260925 15:10:34.503788 20894 raft_consensus.cc:2781] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:34.503971 20892 raft_consensus.cc:2726] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:10:34.504344 20894 raft_consensus.cc:686] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 LEADER]: Becoming Leader. State: Replica: 7857a4cf00eb466dab5f1debf7479d97, State: Running, Role: LEADER
I20260925 15:10:34.504777 20894 consensus_queue.cc:237] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:34.508926 20412 catalog_manager.cc:5495] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "7857a4cf00eb466dab5f1debf7479d97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } health_report { overall_health: UNKNOWN } } }
I20260925 15:10:34.566226 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.560934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60116 (local address 127.4.205.194:46881)
0925 15:10:34.561112 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.561118 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.561136 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:10:34.561387 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.561391 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.561462 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.561645 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:34.562037 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.562775 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.564144 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.564557 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.564688 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.565171 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:34.565638 (+ 467us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:10:34.565784 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.565875 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.565951 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:10:34.573244 20823 raft_consensus.cc:1261] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Refusing update from remote peer 7857a4cf00eb466dab5f1debf7479d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:34.573297 20556 raft_consensus.cc:1261] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Refusing update from remote peer 7857a4cf00eb466dab5f1debf7479d97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:34.574070 20900 consensus_queue.cc:1035] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:34.574435 20894 consensus_queue.cc:1035] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:34.582639 20903 mvcc.cc:205] Tried to move back new op lower bound from 7333269645603483648 to 7333269645335298048. Current Snapshot: MvccSnapshot[applied={T|T < 7333269645603483648}]
I20260925 15:10:34.585835 20904 mvcc.cc:205] Tried to move back new op lower bound from 7333269645603483648 to 7333269645335298048. Current Snapshot: MvccSnapshot[applied={T|T < 7333269645603483648}]
I20260925 15:10:34.595062 20412 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:34.595335 20412 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52438:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:34.599502 20412 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Sending DeleteTablet for 3 replicas of tablet 07df0513fb704de2b84239826c73e33e
I20260925 15:10:34.600620 20670 tablet_service.cc:1514] Processing DeleteTablet for tablet 07df0513fb704de2b84239826c73e33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:34 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60100
I20260925 15:10:34.600790 20803 tablet_service.cc:1514] Processing DeleteTablet for tablet 07df0513fb704de2b84239826c73e33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:34 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:39698
I20260925 15:10:34.601549 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20473
I20260925 15:10:34.603684 20536 tablet_service.cc:1514] Processing DeleteTablet for tablet 07df0513fb704de2b84239826c73e33e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:34 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:48084
I20260925 15:10:34.627665 20915 tablet_replica.cc:330] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: stopping tablet replica
I20260925 15:10:34.629141 20915 raft_consensus.cc:2229] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:34.632107 20915 raft_consensus.cc:2258] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:34.637229 20915 ts_tablet_manager.cc:1905] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:34.639186 20912 tablet_replica.cc:330] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: stopping tablet replica
I20260925 15:10:34.639607 20912 raft_consensus.cc:2229] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:10:34.640195 20912 raft_consensus.cc:2258] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:34.642445 20912 ts_tablet_manager.cc:1905] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:34.650005 20913 tablet_replica.cc:330] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: stopping tablet replica
I20260925 15:10:34.650439 20913 raft_consensus.cc:2229] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:34.651329 20912 ts_tablet_manager.cc:1918] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:34.651513 20912 log.cc:1196] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/07df0513fb704de2b84239826c73e33e
I20260925 15:10:34.652129 20913 raft_consensus.cc:2258] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:34.652173 20912 ts_tablet_manager.cc:1939] T 07df0513fb704de2b84239826c73e33e P 7857a4cf00eb466dab5f1debf7479d97: Deleting consensus metadata
Sep 25 15:10:34 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:10:34.653757 20399 catalog_manager.cc:4841] TS 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881): tablet 07df0513fb704de2b84239826c73e33e (table test-table [id=53e39f83ebd34d46aea75ca294781236]) successfully deleted
I20260925 15:10:34.655136 20915 ts_tablet_manager.cc:1918] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:34.655532 20913 ts_tablet_manager.cc:1905] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:34.655697 20915 log.cc:1196] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/07df0513fb704de2b84239826c73e33e
I20260925 15:10:34.656304 20915 ts_tablet_manager.cc:1939] T 07df0513fb704de2b84239826c73e33e P b9d1a4c26b3949af8ababf853ee67cad: Deleting consensus metadata
I20260925 15:10:34.658465 20396 catalog_manager.cc:4841] TS b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887): tablet 07df0513fb704de2b84239826c73e33e (table test-table [id=53e39f83ebd34d46aea75ca294781236]) successfully deleted
I20260925 15:10:34.662091 20883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.602607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48100 (local address 127.4.205.193:37887)
0925 15:10:34.602761 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.602767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.628311 (+ 25544us) server_negotiation.cc:407] Connection header received
0925 15:10:34.629416 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.629423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.629509 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.629612 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:34.632584 (+ 2972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.633716 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.640578 (+ 6862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.641147 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.641296 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.654590 (+ 13294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:34.654609 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:34.654628 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:34.654675 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:34.658512 (+ 3837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.659429 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.659438 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.659442 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.659620 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.660168 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.660174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.660177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.660611 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:34.660912 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.661284 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.661529 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:10:34.662479 20913 ts_tablet_manager.cc:1918] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:34.662681 20913 log.cc:1196] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/07df0513fb704de2b84239826c73e33e
I20260925 15:10:34.662762 20596 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:34.663280 20913 ts_tablet_manager.cc:1939] T 07df0513fb704de2b84239826c73e33e P 2102353d0f3746fe9908cdbd15be6619: Deleting consensus metadata
I20260925 15:10:34.665918 20399 catalog_manager.cc:4841] TS 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053): tablet 07df0513fb704de2b84239826c73e33e (table test-table [id=53e39f83ebd34d46aea75ca294781236]) successfully deleted
I20260925 15:10:34.770323 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20473
I20260925 15:10:34.788513 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20607
Sep 25 15:10:34 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:10:34.804936 20881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.789314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60122 (local address 127.4.205.194:46881)
0925 15:10:34.789532 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.789537 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.789571 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:10:34.789742 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.789748 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.789830 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.789953 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:34.791322 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.792386 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.793799 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.794474 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.794605 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.798612 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:34.798624 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:34.798641 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:34.798675 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:34.802359 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.802942 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.802951 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.802954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.803113 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.803441 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.803446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.803449 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.803892 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:34.804162 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.804357 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.804586 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:10:34.805696 20730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:34.918308 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20607
I20260925 15:10:34.936383 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20740
Sep 25 15:10:34 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:10:34.950604 20885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:34.937148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.4.205.195:34053)
0925 15:10:34.937342 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:34.937350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:34.937369 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:10:34.937596 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:34.937605 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:34.937673 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:34.937799 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:34.939024 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.939685 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.941045 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:34.941490 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:34.941664 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:34.945858 (+ 4194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:34.945872 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:34.945888 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:34.945918 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:34.948453 (+ 2535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.949031 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.949038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.949040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.949155 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:34.949453 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:34.949456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:34.949458 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:34.949822 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:34.950041 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:34.950235 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 15:10:34.950388 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:10:34.951243 20863 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:35.037360 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20740
I20260925 15:10:35.053761 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20380
I20260925 15:10:35.054914 20442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:35.162351 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20380
I20260925 15:10:55.183120 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--webserver_interface=127.4.205.254
--webserver_port=33593
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:10:55.490286 20920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:55.490520 20920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:55.490599 20920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:55.497164 20920 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:10:55.497263 20920 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:10:55.497314 20920 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:10:55.497375 20920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:10:55.497421 20920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:55.497475 20920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:55.497514 20920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:10:55.497570 20920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:10:55.506346 20920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=33593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:55.507409 20920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:55.508560 20920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:55.530143 20926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:55.530177 20927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:55.530488 20920 server_base.cc:1034] running on GCE node
W20260925 15:10:55.530202 20929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:55.531564 20920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:55.532972 20920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:55.534198 20920 hybrid_clock.cc:648] HybridClock initialized: now 1790349055534127 us; error 45 us; skew 500 ppm
I20260925 15:10:55.534544 20920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:55 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349055, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:55.543859 20920 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:10:55.545377 20920 webserver.cc:460] Webserver started at http://127.4.205.254:33593/ using document root <none> and password file <none>
I20260925 15:10:55.546125 20920 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:55.546223 20920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:55.551944 20920 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:10:55.554659 20936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:55.555974 20920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:55.556178 20920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "6bd8362500b94bc3bc8be6bab840cc16"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "d3e402d0226308cf60d9d79110d137f6"
server_key_iv: "915ca3acad01d4f7b5501c57235eb17c"
server_key_version: "encryptionkey@0"
I20260925 15:10:55.557193 20920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:55.578513 20920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:55.582695 20920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:55.583037 20920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:55.601295 20920 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44489
I20260925 15:10:55.601317 20987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44489 every 8 connection(s)
I20260925 15:10:55.602821 20920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:10:55.604820 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20920
I20260925 15:10:55.605860 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:37887
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=44117
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:10:55.612761 20988 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:10:55.616804 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap starting.
I20260925 15:10:55.636096 20988 log.cc:824] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:55.647439 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:10:55.647895 20988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap complete.
I20260925 15:10:55.657158 20988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:55.658044 20988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Initialized, Role: FOLLOWER
I20260925 15:10:55.658625 20988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:55.658850 20988 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:10:55.658959 20988 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:10:55.659168 20988 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:10:55.663105 20988 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:55.663597 20988 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd8362500b94bc3bc8be6bab840cc16; no voters:
I20260925 15:10:55.664204 20988 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:10:55.664331 20992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:10:55.665800 20992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 LEADER]: Becoming Leader. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Running, Role: LEADER
I20260925 15:10:55.666416 20992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:10:55.666601 20988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:10:55.668776 20994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd8362500b94bc3bc8be6bab840cc16. Latest consensus state: current_term: 2 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:10:55.669100 20994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:55.669646 20993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:10:55.669941 20993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:10:55.678885 20999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:10:55.688795 20999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:10:55.690732 20999 catalog_manager.cc:1259] Loaded cluster ID: 2883f45bce434fecad4ed92e9759f84c
I20260925 15:10:55.690930 20999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:10:55.696231 20999 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:10:55.698283 20999 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Loaded TSK: 0
I20260925 15:10:55.746083 20999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Generated new TSK 1
I20260925 15:10:55.746477 20999 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:10:55.951718 20990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:55.951969 20990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:55.952098 20990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:55.958680 20990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:55.958777 20990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:55.958956 20990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:10:55.967967 20990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:37887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=44117
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:55.969532 20990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:55.971285 20990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:55.983398 21015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:55.983422 21016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:55.984144 21018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:55.985216 20990 server_base.cc:1034] running on GCE node
I20260925 15:10:55.985853 20990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:55.986884 20990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:55.988101 20990 hybrid_clock.cc:648] HybridClock initialized: now 1790349055988048 us; error 50 us; skew 500 ppm
I20260925 15:10:55.988405 20990 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:55 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349055, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:55.991559 20990 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:10:55.993033 20990 webserver.cc:460] Webserver started at http://127.4.205.193:44117/ using document root <none> and password file <none>
I20260925 15:10:55.993764 20990 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:55.993919 20990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:55.999207 20990 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:10:56.001946 21025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.003279 20990 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:10:56.003432 20990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "b9d1a4c26b3949af8ababf853ee67cad"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "41ae06e97a76ac5c8e5d9aa321bd3d26"
server_key_iv: "31df09106cb252f1c7b28e6ed4553c53"
server_key_version: "encryptionkey@0"
I20260925 15:10:56.004345 20990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:56.036427 20990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:56.040568 20990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:56.040838 20990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:56.041981 20990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:56.043277 20990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:56.043365 20990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.043504 20990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:56.043598 20990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.075930 20990 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:37887
I20260925 15:10:56.076009 21137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:37887 every 8 connection(s)
I20260925 15:10:56.077384 20990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:10:56.080941 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 20990
I20260925 15:10:56.081700 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:46881
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=41565
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:10:56 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349055, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:56.100248 21141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.080404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:50759)
0925 15:10:56.081266 (+ 862us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:56.081310 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:56.083090 (+ 1780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:56.084580 (+ 1490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:56.084587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:56.085056 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:56.085617 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:56.085650 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.087986 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.087990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:56.089110 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:56.089121 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.090164 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.090171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:56.090414 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:56.091064 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:56.091086 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:56.093041 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:56.096221 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:56.096230 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:56.096243 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:56.096630 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:56.097068 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:56.097074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:56.097079 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:56.097255 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:56.097990 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:56.097997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:56.098293 (+ 296us) client_negotiation.cc:769] Sending connection context
0925 15:10:56.098537 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 15:10:56.099031 (+ 494us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":376,"threads_started":1}
I20260925 15:10:56.100250 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.080090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:50759 (local address 127.4.205.254:44489)
0925 15:10:56.080764 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:56.080779 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:56.081451 (+ 672us) server_negotiation.cc:407] Connection header received
0925 15:10:56.083685 (+ 2234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:56.083704 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:56.084111 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:56.084482 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:56.085815 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.087841 (+ 2026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.089247 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.089943 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.090131 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:56.093299 (+ 3168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:56.093307 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:56.093319 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:56.093358 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:56.096066 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:56.096769 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:56.096782 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:56.096786 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:56.096945 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:56.097384 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:56.097392 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:56.097399 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:56.097858 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:56.098059 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:56.098727 (+ 668us) server_negotiation.cc:299] Negotiation successful
0925 15:10:56.099096 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":408,"thread_start_us":266,"threads_started":1}
I20260925 15:10:56.103235 21138 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:56.103570 21138 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:56.104447 21138 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:56.106801 20953 ts_manager.cc:194] Registered new tserver with Master: b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887)
I20260925 15:10:56.108736 20953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:50759
W20260925 15:10:56.391294 21142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:56.391534 21142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:56.391630 21142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:56.401252 21142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:56.401371 21142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:56.401546 21142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:10:56.411238 21142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:46881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=41565
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:56.412371 21142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:56.413497 21142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:56.425870 21147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:56.425925 21150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:56.425949 21148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:56.426647 21142 server_base.cc:1034] running on GCE node
I20260925 15:10:56.427167 21142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:56.427932 21142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:56.429114 21142 hybrid_clock.cc:648] HybridClock initialized: now 1790349056429084 us; error 29 us; skew 500 ppm
I20260925 15:10:56.429397 21142 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:56 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:56.432403 21142 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:10:56.433773 21142 webserver.cc:460] Webserver started at http://127.4.205.194:41565/ using document root <none> and password file <none>
I20260925 15:10:56.434376 21142 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:56.434538 21142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:56.439251 21142 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:10:56.441602 21157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.442768 21142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:10:56.442927 21142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "7857a4cf00eb466dab5f1debf7479d97"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "f9a705ba1b5f352dd29804d11166eefa"
server_key_iv: "8a1db3a1d7de109db38d39185e809326"
server_key_version: "encryptionkey@0"
I20260925 15:10:56.443848 21142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:56.489913 21142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:56.494197 21142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:56.494480 21142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:56.495447 21142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:56.496701 21142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:56.496860 21142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.496994 21142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:56.497090 21142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.529381 21142 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:46881
I20260925 15:10:56.529464 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:46881 every 8 connection(s)
I20260925 15:10:56.530884 21142 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:10:56.534404 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21142
I20260925 15:10:56.535122 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:34053
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=41205
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:10:56 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:56.553880 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.533368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:60277 (local address 127.4.205.254:44489)
0925 15:10:56.533548 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:56.533553 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:56.534935 (+ 1382us) server_negotiation.cc:407] Connection header received
0925 15:10:56.537831 (+ 2896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:56.537835 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:56.537899 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:56.537977 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:56.539974 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.540681 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.543135 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.543681 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.543816 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:56.547066 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:56.547075 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:56.547083 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:56.547129 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:56.550716 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:56.551393 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:56.551401 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:56.551408 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:56.551555 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:56.551951 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:56.551958 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:56.551964 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:56.552329 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:56.552517 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:56.553108 (+ 591us) server_negotiation.cc:299] Negotiation successful
0925 15:10:56.553369 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:10:56.554354 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.533925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:60277)
0925 15:10:56.534745 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:56.534804 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:56.537513 (+ 2709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:56.538234 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:56.538247 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:56.539015 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:56.539797 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:56.539825 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.540850 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.540859 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:56.542989 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:56.543006 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:56.543887 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:56.543891 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:56.544075 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:56.544737 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:56.544757 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:56.546852 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:56.550840 (+ 3988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:56.550847 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:56.550859 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:56.551256 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:56.551658 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:56.551662 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:56.551664 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:56.551822 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:56.552474 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:56.552478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:56.552755 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 15:10:56.552958 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 15:10:56.553268 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":291,"threads_started":1}
I20260925 15:10:56.555742 21270 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:56.556023 21270 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:56.556672 21270 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:56.558020 20953 ts_manager.cc:194] Registered new tserver with Master: 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881)
I20260925 15:10:56.558820 20953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:60277
W20260925 15:10:56.874042 21273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:10:56.874450 21273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:10:56.874608 21273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:10:56.881501 21273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:10:56.881659 21273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:10:56.881848 21273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:10:56.890750 21273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=41205
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:10:56.891816 21273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:10:56.892935 21273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:10:56.906153 21278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:56.906261 21279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:10:56.906286 21281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:10:56.907104 21273 server_base.cc:1034] running on GCE node
I20260925 15:10:56.907855 21273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:10:56.908759 21273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:10:56.909941 21273 hybrid_clock.cc:648] HybridClock initialized: now 1790349056909882 us; error 55 us; skew 500 ppm
I20260925 15:10:56.910193 21273 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:10:56 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:10:56.913142 21273 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:10:56.914477 21273 webserver.cc:460] Webserver started at http://127.4.205.195:41205/ using document root <none> and password file <none>
I20260925 15:10:56.915146 21273 fs_manager.cc:359] Metadata directory not provided
I20260925 15:10:56.915283 21273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:10:56.920220 21273 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260925 15:10:56.922838 21288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.924003 21273 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:10:56.924161 21273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "2102353d0f3746fe9908cdbd15be6619"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "5f1588ff6248fcdfc39cdaa7e8e70c22"
server_key_iv: "aed2a890b7f13aa3282d104f2669528f"
server_key_version: "encryptionkey@0"
I20260925 15:10:56.925047 21273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:10:56.951182 21273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:10:56.954779 21273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:10:56.955029 21273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:10:56.955921 21273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:10:56.957180 21273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:10:56.957288 21273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.957435 21273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:10:56.957525 21273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:10:56.991183 21273 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34053
I20260925 15:10:56.991291 21400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34053 every 8 connection(s)
I20260925 15:10:56.992635 21273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:10:57.002270 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21273
Sep 25 15:10:57 dist-test-slave-64fx krb5kdc[20367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349056, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:10:57.013254 21140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.995103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:33199 (local address 127.4.205.254:44489)
0925 15:10:56.995279 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:56.995283 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:56.996339 (+ 1056us) server_negotiation.cc:407] Connection header received
0925 15:10:56.998200 (+ 1861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:56.998205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:56.998300 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:56.998407 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:56.999808 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.000519 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.002265 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.002724 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.002882 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.006899 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.006904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.006907 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.006936 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.010151 (+ 3215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.010971 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.010979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.010984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.011125 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.011525 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.011531 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.011535 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.011909 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.012066 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.012679 (+ 613us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.012890 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:10:57.014225 21403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:56.995514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:33199)
0925 15:10:56.996158 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:56.996221 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:56.997951 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:56.998631 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:56.998639 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:56.999115 (+ 476us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:10:56.999645 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:56.999661 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.000678 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.000682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.002095 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.002112 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.002949 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.002959 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.003210 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.004074 (+ 864us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:10:57.004096 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:10:57.006708 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:10:57.010297 (+ 3589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:57.010302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:57.010313 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:57.010851 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:57.011234 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:10:57.011237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:10:57.011239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:10:57.011405 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:10:57.012046 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:10:57.012051 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:10:57.012322 (+ 271us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.012548 (+ 226us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.012883 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":253,"threads_started":1}
I20260925 15:10:57.015745 21401 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44489
I20260925 15:10:57.016021 21401 heartbeater.cc:464] Registering TS with master...
I20260925 15:10:57.016670 21401 heartbeater.cc:511] Master 127.4.205.254:44489 requested a full tablet report, sending...
I20260925 15:10:57.018177 20953 ts_manager.cc:194] Registered new tserver with Master: 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053)
I20260925 15:10:57.019284 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.003136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45978 (local address 127.4.205.254:44489)
0925 15:10:57.003672 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.003677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.004454 (+ 777us) server_negotiation.cc:407] Connection header received
0925 15:10:57.004557 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.004563 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.004646 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.004757 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:57.006444 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.007345 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.009377 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.010015 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.010109 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.013117 (+ 3008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.013126 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.013133 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.013167 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.016506 (+ 3339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.017196 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.017208 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.017211 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.017357 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.017807 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.017812 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.017814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.018125 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.018387 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.018699 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.018913 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":272,"threads_started":1}
I20260925 15:10:57.019366 20953 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:33199
I20260925 15:10:57.021785 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:10:57.027503 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.022616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45994 (local address 127.4.205.254:44489)
0925 15:10:57.022765 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.022769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.023214 (+ 445us) server_negotiation.cc:407] Connection header received
0925 15:10:57.023367 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.023370 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.023477 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.023581 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:57.023916 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.024526 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.025998 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.026391 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.026477 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.026932 (+ 455us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:57.027088 (+ 156us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:10:57.027322 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45994: token expired
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:10:57.027619 21404 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45994: token expired
W20260925 15:10:57.028128 20874 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.4.205.254:44489)
I20260925 15:10:57.039608 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.028561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46004 (local address 127.4.205.254:44489)
0925 15:10:57.028721 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.028745 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.028811 (+ 66us) server_negotiation.cc:407] Connection header received
0925 15:10:57.029007 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.029012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.029070 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.029164 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:57.030530 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.031175 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.032386 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.032758 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.032872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.035047 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.035053 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.035058 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.035093 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.037142 (+ 2049us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.037830 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.037838 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.037843 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.037987 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.038347 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.038352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.038358 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.038609 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.038843 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.039143 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.039340 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:10:57.042497 20952 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:10:57.049539 20952 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:10:57.072014 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.059674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46881 (local address 127.0.0.1:44818)
0925 15:10:57.060622 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:57.060690 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:57.060898 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:57.062308 (+ 1410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:57.062318 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:57.062368 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:57.062902 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.062916 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.065408 (+ 2492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.065416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.068349 (+ 2933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.068374 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.070982 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.070990 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.071218 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.071274 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.071404 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.071603 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":701,"threads_started":1}
I20260925 15:10:57.073680 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.059847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.4.205.194:46881)
0925 15:10:57.061595 (+ 1748us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.061607 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.061642 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:10:57.061774 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.061785 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.062047 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.063128 (+ 1081us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:57.063404 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.065235 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.068645 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.070698 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.072688 (+ 1990us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.072936 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.073039 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.073241 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1447,"threads_started":1}
I20260925 15:10:57.077623 21205 tablet_service.cc:1467] Processing CreateTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 (DEFAULT_TABLE table=test-table [id=9aa299f31efd4a6db44707678889642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:57.078938 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42728cbd25c04b42b5f5b3653ed195f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:57.082031 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.061252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34053 (local address 127.0.0.1:56662)
0925 15:10:57.072215 (+ 10963us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:57.072230 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:57.072308 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:57.075519 (+ 3211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:57.075522 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:57.075541 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:57.075719 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.075730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.077793 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.077798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.079498 (+ 1700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.079511 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.081263 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.081269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.081466 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.081504 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.081614 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.081747 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10884,"thread_start_us":10534,"threads_started":1}
I20260925 15:10:57.082708 21415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.060154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:37887 (local address 127.0.0.1:54918)
0925 15:10:57.072042 (+ 11888us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:57.072060 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:57.072163 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:57.073639 (+ 1476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:57.073646 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:57.073679 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:57.074086 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.074098 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.076464 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.076470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.079432 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.079447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.081617 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.081624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.081801 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.081831 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.081912 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.082036 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11762,"mutex_wait_us":1,"thread_start_us":14254,"threads_started":1}
I20260925 15:10:57.083520 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.060555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54918 (local address 127.4.205.193:37887)
0925 15:10:57.072701 (+ 12146us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.072708 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.072731 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:10:57.072820 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.072828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.073105 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.073318 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:57.074889 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.076320 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.079627 (+ 3307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.081051 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.081195 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.081450 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.082919 (+ 1469us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.083141 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12010,"thread_start_us":14142,"threads_started":1}
I20260925 15:10:57.083827 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.061872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.4.205.195:34053)
0925 15:10:57.075024 (+ 13152us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.075031 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.075058 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:10:57.075135 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.075141 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.075320 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.075873 (+ 553us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:57.076086 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.077652 (+ 1566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.079684 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.081046 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.082338 (+ 1292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.083201 (+ 863us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.083272 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.083443 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12980,"thread_start_us":13075,"threads_started":1}
I20260925 15:10:57.087188 21336 tablet_service.cc:1467] Processing CreateTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 (DEFAULT_TABLE table=test-table [id=9aa299f31efd4a6db44707678889642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:57.087188 21073 tablet_service.cc:1467] Processing CreateTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 (DEFAULT_TABLE table=test-table [id=9aa299f31efd4a6db44707678889642c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:10:57.088451 21336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42728cbd25c04b42b5f5b3653ed195f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:57.088457 21073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42728cbd25c04b42b5f5b3653ed195f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:10:57.092356 21417 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Bootstrap starting.
I20260925 15:10:57.095567 21417 tablet_bootstrap.cc:654] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:57.096537 21417 log.cc:824] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:57.098798 21417 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: No bootstrap required, opened a new log
I20260925 15:10:57.099068 21417 ts_tablet_manager.cc:1397] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:10:57.102407 21419 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Bootstrap starting.
I20260925 15:10:57.103880 21420 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Bootstrap starting.
I20260925 15:10:57.105633 21419 tablet_bootstrap.cc:654] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:57.106591 21420 tablet_bootstrap.cc:654] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Neither blocks nor log segments found. Creating new log.
I20260925 15:10:57.106737 21419 log.cc:824] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:57.107672 21420 log.cc:824] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Log is configured to *not* fsync() on all Append() calls
I20260925 15:10:57.108773 21417 raft_consensus.cc:348] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.109148 21417 raft_consensus.cc:374] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:57.109243 21417 raft_consensus.cc:729] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7857a4cf00eb466dab5f1debf7479d97, State: Initialized, Role: FOLLOWER
I20260925 15:10:57.110060 21419 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: No bootstrap required, opened a new log
I20260925 15:10:57.110132 21420 tablet_bootstrap.cc:492] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: No bootstrap required, opened a new log
I20260925 15:10:57.109959 21417 consensus_queue.cc:260] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.110409 21420 ts_tablet_manager.cc:1397] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 15:10:57.110441 21419 ts_tablet_manager.cc:1397] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:10:57.111028 21270 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
I20260925 15:10:57.111616 21417 ts_tablet_manager.cc:1428] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20260925 15:10:57.112147 21138 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
I20260925 15:10:57.118863 21420 raft_consensus.cc:348] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.119148 21420 raft_consensus.cc:374] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:57.119272 21420 raft_consensus.cc:729] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2102353d0f3746fe9908cdbd15be6619, State: Initialized, Role: FOLLOWER
I20260925 15:10:57.119788 21420 consensus_queue.cc:260] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.119797 21419 raft_consensus.cc:348] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.120178 21419 raft_consensus.cc:374] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:10:57.120270 21419 raft_consensus.cc:729] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9d1a4c26b3949af8ababf853ee67cad, State: Initialized, Role: FOLLOWER
I20260925 15:10:57.120767 21401 heartbeater.cc:503] Master 127.4.205.254:44489 was elected leader, sending a full tablet report...
I20260925 15:10:57.120782 21419 consensus_queue.cc:260] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.122370 21419 ts_tablet_manager.cc:1428] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:10:57.122479 21420 ts_tablet_manager.cc:1428] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20260925 15:10:57.244307 21402 tablet.cc:2369] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:10:57.282804 21271 tablet.cc:2369] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:10:57.329788 21139 tablet.cc:2369] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:10:57.484287 21425 raft_consensus.cc:482] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:10:57.484588 21425 raft_consensus.cc:504] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.486217 21425 leader_election.cc:288] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881), 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053)
I20260925 15:10:57.492259 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.486678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:57263 (local address 127.4.205.194:46881)
0925 15:10:57.486939 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.486945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.487265 (+ 320us) server_negotiation.cc:407] Connection header received
0925 15:10:57.487514 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.487521 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.487610 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.487707 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:57.488277 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.488955 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.490680 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.491410 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.491602 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.491670 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.491912 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.492026 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 15:10:57.492323 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.486493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46881 (local address 127.4.205.193:57263)
0925 15:10:57.487126 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:57.487143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:57.487352 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:57.487737 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:57.487743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:57.487782 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:57.488125 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.488133 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.489107 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.489113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.490518 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.490530 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.491616 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.491621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.491765 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.491786 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.491888 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.492006 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":365,"threads_started":1}
I20260925 15:10:57.492655 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.487387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:43765 (local address 127.4.205.195:34053)
0925 15:10:57.487618 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.487625 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.488034 (+ 409us) server_negotiation.cc:407] Connection header received
0925 15:10:57.488239 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.488242 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.488299 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.488389 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:10:57.488894 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.489507 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.491090 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.491827 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.491990 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.492056 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.492316 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.492433 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:10:57.492710 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.487245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:34053 (local address 127.4.205.193:43765)
0925 15:10:57.487927 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0925 15:10:57.487947 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:10:57.488086 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:10:57.488415 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:10:57.488420 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:10:57.488442 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:10:57.488725 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.488733 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.489651 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.489658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.490941 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:10:57.490952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.492020 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.492025 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:10:57.492178 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.492199 (+ 21us) client_negotiation.cc:769] Sending connection context
0925 15:10:57.492291 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:10:57.492413 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":354,"threads_started":1}
I20260925 15:10:57.493314 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42728cbd25c04b42b5f5b3653ed195f6" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7857a4cf00eb466dab5f1debf7479d97" is_pre_election: true
I20260925 15:10:57.493613 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42728cbd25c04b42b5f5b3653ed195f6" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2102353d0f3746fe9908cdbd15be6619" is_pre_election: true
I20260925 15:10:57.493860 21225 raft_consensus.cc:2454] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d1a4c26b3949af8ababf853ee67cad in term 0.
I20260925 15:10:57.494065 21356 raft_consensus.cc:2454] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b9d1a4c26b3949af8ababf853ee67cad in term 0.
I20260925 15:10:57.494506 21029 leader_election.cc:302] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7857a4cf00eb466dab5f1debf7479d97, b9d1a4c26b3949af8ababf853ee67cad; no voters:
I20260925 15:10:57.494779 21425 raft_consensus.cc:2781] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:10:57.494918 21425 raft_consensus.cc:482] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:10:57.495026 21425 raft_consensus.cc:3037] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:57.497064 21425 raft_consensus.cc:504] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.497663 21425 leader_election.cc:288] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 election: Requested vote from peers 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881), 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053)
I20260925 15:10:57.498067 21225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42728cbd25c04b42b5f5b3653ed195f6" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7857a4cf00eb466dab5f1debf7479d97"
I20260925 15:10:57.498153 21356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42728cbd25c04b42b5f5b3653ed195f6" candidate_uuid: "b9d1a4c26b3949af8ababf853ee67cad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2102353d0f3746fe9908cdbd15be6619"
I20260925 15:10:57.498266 21225 raft_consensus.cc:3037] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:57.498364 21356 raft_consensus.cc:3037] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:10:57.500262 21225 raft_consensus.cc:2454] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d1a4c26b3949af8ababf853ee67cad in term 1.
I20260925 15:10:57.500274 21356 raft_consensus.cc:2454] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b9d1a4c26b3949af8ababf853ee67cad in term 1.
I20260925 15:10:57.500597 21029 leader_election.cc:302] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2102353d0f3746fe9908cdbd15be6619, b9d1a4c26b3949af8ababf853ee67cad; no voters:
I20260925 15:10:57.500885 21425 raft_consensus.cc:2781] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:10:57.501436 21425 raft_consensus.cc:686] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 1 LEADER]: Becoming Leader. State: Replica: b9d1a4c26b3949af8ababf853ee67cad, State: Running, Role: LEADER
I20260925 15:10:57.501893 21425 consensus_queue.cc:237] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } }
I20260925 15:10:57.506120 20953 catalog_manager.cc:5495] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad reported cstate change: term changed from 0 to 1, leader changed from <none> to b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193). New cstate: current_term: 1 leader_uuid: "b9d1a4c26b3949af8ababf853ee67cad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b9d1a4c26b3949af8ababf853ee67cad" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37887 } health_report { overall_health: HEALTHY } } }
I20260925 15:10:57.532615 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.527138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54926 (local address 127.4.205.193:37887)
0925 15:10:57.527320 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.527327 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.527345 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:10:57.527582 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.527585 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.527650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.527797 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:10:57.528117 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.528865 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.530744 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.531079 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.531160 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.531708 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:10:57.532123 (+ 415us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:10:57.532215 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.532352 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.532404 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:10:57.540356 21356 raft_consensus.cc:1261] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Refusing update from remote peer b9d1a4c26b3949af8ababf853ee67cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:57.540483 21225 raft_consensus.cc:1261] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Refusing update from remote peer b9d1a4c26b3949af8ababf853ee67cad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:10:57.541287 21428 consensus_queue.cc:1035] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2102353d0f3746fe9908cdbd15be6619" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 34053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:57.541736 21425 consensus_queue.cc:1035] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [LEADER]: Connected to new peer: Peer: permanent_uuid: "7857a4cf00eb466dab5f1debf7479d97" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:10:57.565376 20952 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:57.565635 20952 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:10:57.568768 20952 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Sending DeleteTablet for 3 replicas of tablet 42728cbd25c04b42b5f5b3653ed195f6
I20260925 15:10:57.569841 21205 tablet_service.cc:1514] Processing DeleteTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:57 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:44818
I20260925 15:10:57.570112 21073 tablet_service.cc:1514] Processing DeleteTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:57 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:54918
I20260925 15:10:57.570693 21441 tablet_replica.cc:330] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: stopping tablet replica
I20260925 15:10:57.570897 21336 tablet_service.cc:1514] Processing DeleteTablet for tablet 42728cbd25c04b42b5f5b3653ed195f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:10:57 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:56662
I20260925 15:10:57.571118 21441 raft_consensus.cc:2229] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:57.571587 21441 raft_consensus.cc:2258] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:57.572176 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20990
I20260925 15:10:57.580624 21441 ts_tablet_manager.cc:1905] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:57.582233 21443 tablet_replica.cc:330] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: stopping tablet replica
I20260925 15:10:57.582659 21443 raft_consensus.cc:2229] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:10:57.583281 21443 raft_consensus.cc:2258] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:57.583977 21442 tablet_replica.cc:330] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: stopping tablet replica
I20260925 15:10:57.584295 21442 raft_consensus.cc:2229] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:10:57.584924 21442 raft_consensus.cc:2258] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:10:57.586599 21443 ts_tablet_manager.cc:1905] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:57.587065 21442 ts_tablet_manager.cc:1905] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:10:57.592010 21441 ts_tablet_manager.cc:1918] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:57.593123 21443 ts_tablet_manager.cc:1918] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:57.593305 21443 log.cc:1196] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/42728cbd25c04b42b5f5b3653ed195f6
I20260925 15:10:57.593856 21443 ts_tablet_manager.cc:1939] T 42728cbd25c04b42b5f5b3653ed195f6 P 2102353d0f3746fe9908cdbd15be6619: Deleting consensus metadata
I20260925 15:10:57.594631 21441 log.cc:1196] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/42728cbd25c04b42b5f5b3653ed195f6
I20260925 15:10:57.594954 21442 ts_tablet_manager.cc:1918] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:10:57.595196 21442 log.cc:1196] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/42728cbd25c04b42b5f5b3653ed195f6
I20260925 15:10:57.595909 20940 catalog_manager.cc:4841] TS 2102353d0f3746fe9908cdbd15be6619 (127.4.205.195:34053): tablet 42728cbd25c04b42b5f5b3653ed195f6 (table test-table [id=9aa299f31efd4a6db44707678889642c]) successfully deleted
I20260925 15:10:57.596233 21442 ts_tablet_manager.cc:1939] T 42728cbd25c04b42b5f5b3653ed195f6 P b9d1a4c26b3949af8ababf853ee67cad: Deleting consensus metadata
I20260925 15:10:57.597322 21441 ts_tablet_manager.cc:1939] T 42728cbd25c04b42b5f5b3653ed195f6 P 7857a4cf00eb466dab5f1debf7479d97: Deleting consensus metadata
I20260925 15:10:57.597941 20937 catalog_manager.cc:4841] TS b9d1a4c26b3949af8ababf853ee67cad (127.4.205.193:37887): tablet 42728cbd25c04b42b5f5b3653ed195f6 (table test-table [id=9aa299f31efd4a6db44707678889642c]) successfully deleted
I20260925 15:10:57.598959 20940 catalog_manager.cc:4841] TS 7857a4cf00eb466dab5f1debf7479d97 (127.4.205.194:46881): tablet 42728cbd25c04b42b5f5b3653ed195f6 (table test-table [id=9aa299f31efd4a6db44707678889642c]) successfully deleted
I20260925 15:10:57.610273 21414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.572841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54928 (local address 127.4.205.193:37887)
0925 15:10:57.572977 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.572983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.594281 (+ 21298us) server_negotiation.cc:407] Connection header received
0925 15:10:57.594444 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.594449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.594524 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.594625 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:57.596867 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.598158 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.599697 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.600338 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.600476 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.603524 (+ 3048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.603536 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.603550 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.603592 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.607194 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.607981 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.607990 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.607993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.608190 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.608626 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.608631 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.608636 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.609008 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.609353 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.609643 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.609857 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:10:57.610926 21133 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:57.702883 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20990
I20260925 15:10:57.720472 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 21142
I20260925 15:10:57.732928 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.721195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.4.205.194:46881)
0925 15:10:57.721434 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.721439 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.721471 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:10:57.721678 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.721682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.721744 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.721822 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:57.723117 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.723759 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.724937 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.725290 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.725373 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.727905 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.727915 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.727928 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.727956 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.730724 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.731288 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.731294 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.731296 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.731419 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.731679 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.731682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.731683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.732012 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.732214 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.732416 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.732568 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 15:10:57.733502 21265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:57.818400 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 21142
I20260925 15:10:57.833943 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 21273
I20260925 15:10:57.846107 21416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:10:57.834630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56664 (local address 127.4.205.195:34053)
0925 15:10:57.834784 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:10:57.834788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:10:57.834803 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:10:57.834915 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:10:57.834919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:10:57.834974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:10:57.835045 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:10:57.836248 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.836965 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.838425 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:10:57.838794 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:10:57.838874 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:10:57.841271 (+ 2397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:10:57.841283 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:10:57.841298 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:10:57.841329 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:10:57.843907 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.844500 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.844506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.844508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.844630 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:10:57.844909 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:10:57.844911 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:10:57.844913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:10:57.845240 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:10:57.845431 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 15:10:57.845725 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:10:57.845868 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:10:57.846728 21396 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:57.930910 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 21273
I20260925 15:10:57.948029 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 20920
I20260925 15:10:57.949050 20983 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:10:58.059922 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 20920
I20260925 15:11:18.081274 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--webserver_interface=127.4.205.254
--webserver_port=33593
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:18.385717 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:18.385957 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:18.386118 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:18.393599 21449 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:11:18.393677 21449 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:11:18.393760 21449 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:11:18.393833 21449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:18.393937 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:18.394029 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:18.394098 21449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:18.394172 21449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:18.407641 21449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44489
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=33593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:18.408782 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:18.410081 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:18.421512 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:18.421603 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:18.421831 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:18.426836 21449 server_base.cc:1034] running on GCE node
I20260925 15:11:18.427609 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:18.428864 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:18.430142 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790349078430070 us; error 35 us; skew 500 ppm
I20260925 15:11:18.430428 21449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:18 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349078, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:18.433483 21449 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:11:18.435065 21449 webserver.cc:460] Webserver started at http://127.4.205.254:33593/ using document root <none> and password file <none>
I20260925 15:11:18.435760 21449 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:18.435902 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:18.441114 21449 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260925 15:11:18.443809 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:18.445067 21449 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:11:18.445245 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "6bd8362500b94bc3bc8be6bab840cc16"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "d3e402d0226308cf60d9d79110d137f6"
server_key_iv: "915ca3acad01d4f7b5501c57235eb17c"
server_key_version: "encryptionkey@0"
I20260925 15:11:18.446281 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:18.466408 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:18.470299 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:18.470557 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:18.488217 21449 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44489
I20260925 15:11:18.488257 21515 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44489 every 8 connection(s)
I20260925 15:11:18.489645 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:18.492563 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21449
I20260925 15:11:18.493350 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:37887
--local_ip_for_outbound_sockets=127.4.205.193
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=44117
--webserver_interface=127.4.205.193
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:11:18.496088 21516 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:11:18.499445 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap starting.
I20260925 15:11:18.529295 21516 log.cc:824] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:18.549190 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:11:18.549940 21516 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Bootstrap complete.
I20260925 15:11:18.563594 21516 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:11:18.564458 21516 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Initialized, Role: FOLLOWER
I20260925 15:11:18.565099 21516 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:11:18.565344 21516 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:18.565470 21516 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:18.565682 21516 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:11:18.569226 21516 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:11:18.569818 21516 leader_election.cc:302] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6bd8362500b94bc3bc8be6bab840cc16; no voters:
I20260925 15:11:18.570429 21516 leader_election.cc:288] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:11:18.570626 21520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:11:18.571774 21520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [term 3 LEADER]: Becoming Leader. State: Replica: 6bd8362500b94bc3bc8be6bab840cc16, State: Running, Role: LEADER
I20260925 15:11:18.572474 21516 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:18.572418 21520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } }
I20260925 15:11:18.575224 21522 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6bd8362500b94bc3bc8be6bab840cc16. Latest consensus state: current_term: 3 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:11:18.575627 21522 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:18.576207 21521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6bd8362500b94bc3bc8be6bab840cc16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6bd8362500b94bc3bc8be6bab840cc16" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44489 } } }
I20260925 15:11:18.576530 21521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:18.581379 21528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:18.589985 21528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:18.591459 21528 catalog_manager.cc:1259] Loaded cluster ID: 2883f45bce434fecad4ed92e9759f84c
I20260925 15:11:18.591605 21528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:18.596012 21528 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:18.597536 21528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Loaded TSK: 0
I20260925 15:11:18.597755 21528 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6bd8362500b94bc3bc8be6bab840cc16: Loaded TSK: 1
F20260925 15:11:18.598881 21528 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdec4fd09ad google::LogMessage::Fail() at ??:0
@ 0x7fdec4fd388a google::LogMessage::SendToLog() at ??:0
@ 0x7fdec4fd04de google::LogMessage::Flush() at ??:0
@ 0x7fdec4fd1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdec8d347d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fdec8d2373e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fdeda9ff1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fdeda9fe783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fdedaa8a2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fdedaa8a0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fdedaad45e9 std::function<>::operator()() at ??:0
@ 0x7fdedaa02cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fdedaa00c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdedaa88610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fdedaa88439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdec7c1c345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdec7c1f91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fdec7c1f799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fdec7bfaeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdecaed46db start_thread at ??:0
@ 0x7fdec3e6a61f clone at ??:0
W20260925 15:11:18.879446 21518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:18.879703 21518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:18.879860 21518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:18.886601 21518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:18.886716 21518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:18.886895 21518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:18.895604 21518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:37887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=44117
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:18.896631 21518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:18.897778 21518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:18.910169 21563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:18.910197 21566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:18.910118 21564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:18.911265 21518 server_base.cc:1034] running on GCE node
I20260925 15:11:18.911846 21518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:18.912671 21518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:18.913890 21518 hybrid_clock.cc:648] HybridClock initialized: now 1790349078913842 us; error 46 us; skew 500 ppm
I20260925 15:11:18.914152 21518 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:18 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349078, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:18.916987 21518 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:11:18.918387 21518 webserver.cc:460] Webserver started at http://127.4.205.193:44117/ using document root <none> and password file <none>
I20260925 15:11:18.919001 21518 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:18.919140 21518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:18.923938 21518 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:11:18.926396 21573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:18.927469 21518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:11:18.927623 21518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "b9d1a4c26b3949af8ababf853ee67cad"
format_stamp: "Formatted at 2026-09-25 15:10:32 on dist-test-slave-64fx"
server_key: "41ae06e97a76ac5c8e5d9aa321bd3d26"
server_key_iv: "31df09106cb252f1c7b28e6ed4553c53"
server_key_version: "encryptionkey@0"
I20260925 15:11:18.928648 21518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:18.948354 21518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:18.951793 21518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:18.952023 21518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:18.952886 21518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:18.954084 21518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:18.954210 21518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:18.954339 21518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:18.954433 21518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:18.985857 21518 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:37887
I20260925 15:11:18.985929 21685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:37887 every 8 connection(s)
I20260925 15:11:18.987231 21518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:18.989501 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21518
I20260925 15:11:18.990118 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:46881
--local_ip_for_outbound_sockets=127.4.205.194
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=41565
--webserver_interface=127.4.205.194
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:11:18.992051 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:18.990021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:60645)
0925 15:11:18.990912 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:18.991176 (+ 264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":422,"threads_started":1}
W20260925 15:11:18.992766 21686 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:44489 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:44489: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
I20260925 15:11:18.993534 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:18.993230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:38745)
0925 15:11:18.993368 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:18.993443 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:11:18.994313 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:18.994014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:47087)
0925 15:11:18.994138 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:18.994215 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:11:18.994501 21686 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:11:19.312335 21689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:19.312698 21689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:19.312901 21689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:19.325394 21689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:19.325525 21689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:19.325768 21689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:19.341472 21689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:46881
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=41565
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:19.342713 21689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:19.343954 21689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:19.356257 21697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:19.356343 21694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:19.356375 21695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:19.356917 21689 server_base.cc:1034] running on GCE node
I20260925 15:11:19.357518 21689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:19.358464 21689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:19.359648 21689 hybrid_clock.cc:648] HybridClock initialized: now 1790349079359611 us; error 31 us; skew 500 ppm
I20260925 15:11:19.359915 21689 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:19 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:19.362972 21689 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:11:19.364223 21689 webserver.cc:460] Webserver started at http://127.4.205.194:41565/ using document root <none> and password file <none>
I20260925 15:11:19.364779 21689 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:19.364873 21689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:19.369685 21689 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:11:19.372112 21704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.373301 21689 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:11:19.373463 21689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "7857a4cf00eb466dab5f1debf7479d97"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "f9a705ba1b5f352dd29804d11166eefa"
server_key_iv: "8a1db3a1d7de109db38d39185e809326"
server_key_version: "encryptionkey@0"
I20260925 15:11:19.374439 21689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:19.386298 21689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:19.390277 21689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:19.390537 21689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:19.391419 21689 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:19.392661 21689 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:19.392781 21689 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.392901 21689 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:19.392984 21689 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.425853 21689 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:46881
I20260925 15:11:19.425976 21816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:46881 every 8 connection(s)
I20260925 15:11:19.427332 21689 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:19.432430 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.430359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:56735)
0925 15:11:19.431042 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.431269 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":268,"threads_started":1}
I20260925 15:11:19.432502 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21689
W20260925 15:11:19.433233 21817 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:44489 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:44489: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
I20260925 15:11:19.433269 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:34053
--local_ip_for_outbound_sockets=127.4.205.195
--tserver_master_addrs=127.4.205.254:44489
--webserver_port=41205
--webserver_interface=127.4.205.195
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:11:19.434159 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.433755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:52787)
0925 15:11:19.433928 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.434016 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 15:11:19.435052 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.434644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:51271)
0925 15:11:19.434813 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.434906 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:11:19.435283 21817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:11:19.739838 21820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:19.740087 21820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:19.740245 21820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:19.747236 21820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:19.747319 21820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:19.747493 21820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:19.756251 21820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:34053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=41205
--tserver_master_addrs=127.4.205.254:44489
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:19.757316 21820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:19.758452 21820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:19.770375 21825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:19.770375 21828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:19.770417 21826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:19.771130 21820 server_base.cc:1034] running on GCE node
I20260925 15:11:19.771667 21820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:19.772476 21820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:19.773648 21820 hybrid_clock.cc:648] HybridClock initialized: now 1790349079773619 us; error 39 us; skew 500 ppm
I20260925 15:11:19.774011 21820 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:19 dist-test-slave-64fx krb5kdc[20367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349079, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:19.776738 21820 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:11:19.778115 21820 webserver.cc:460] Webserver started at http://127.4.205.195:41205/ using document root <none> and password file <none>
I20260925 15:11:19.778741 21820 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:19.778847 21820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:19.783613 21820 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260925 15:11:19.786020 21835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.787107 21820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:11:19.787271 21820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "2102353d0f3746fe9908cdbd15be6619"
format_stamp: "Formatted at 2026-09-25 15:10:33 on dist-test-slave-64fx"
server_key: "5f1588ff6248fcdfc39cdaa7e8e70c22"
server_key_iv: "aed2a890b7f13aa3282d104f2669528f"
server_key_version: "encryptionkey@0"
I20260925 15:11:19.788168 21820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:19.816743 21820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:19.820439 21820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:19.820705 21820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:19.821655 21820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:19.822875 21820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:19.822989 21820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.823110 21820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:19.823210 21820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:19.854579 21820 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:34053
I20260925 15:11:19.854662 21947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:34053 every 8 connection(s)
I20260925 15:11:19.855878 21820 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:19.860163 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.858492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:32989)
0925 15:11:19.859182 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.859408 (+ 226us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":263,"threads_started":1}
W20260925 15:11:19.860786 21948 heartbeater.cc:644] Failed to heartbeat to 127.4.205.254:44489 (0 consecutive failures): Network error: Failed to ping master at 127.4.205.254:44489: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
I20260925 15:11:19.861524 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.861241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:34553)
0925 15:11:19.861366 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.861428 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:11:19.862352 21950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.862045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:35021)
0925 15:11:19.862169 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.862238 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:11:19.862591 21948 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:11:19.865324 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 21820
I20260925 15:11:19.995731 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:19.994983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:51541)
0925 15:11:19.995409 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:19.995519 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":201,"threads_started":1}
I20260925 15:11:20.436437 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:20.435735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:46435)
0925 15:11:20.436150 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:20.436221 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":210,"threads_started":1}
I20260925 15:11:20.863788 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:20.863043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:48003)
0925 15:11:20.863471 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:20.863574 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":215,"threads_started":1}
I20260925 15:11:20.997334 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:20.996610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:43451)
0925 15:11:20.997107 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:20.997180 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":260,"threads_started":1}
I20260925 15:11:21.438023 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:21.437314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:58725)
0925 15:11:21.437792 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:21.437867 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":202,"threads_started":1}
I20260925 15:11:21.865391 21957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:21.864766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:35853)
0925 15:11:21.865179 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:21.865251 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260925 15:11:21.998775 21958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:21.998107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:49813)
0925 15:11:21.998538 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:21.998611 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":238,"threads_started":1}
I20260925 15:11:22.439508 21959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:22.438852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:43335)
0925 15:11:22.439260 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:22.439345 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20260925 15:11:22.866912 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:22.866292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:42441)
0925 15:11:22.866687 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:22.866758 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":206,"threads_started":1}
I20260925 15:11:23.000275 21961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:22.999589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:39067)
0925 15:11:23.000016 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:23.000104 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":230,"threads_started":1}
I20260925 15:11:23.441098 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:23.440366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:43971)
0925 15:11:23.440762 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:23.440838 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":199,"threads_started":1}
I20260925 15:11:23.868727 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:23.867989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:51577)
0925 15:11:23.868427 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:23.868510 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":215,"threads_started":1}
I20260925 15:11:24.001957 21964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:24.001203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:50673)
0925 15:11:24.001704 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:24.001798 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":218,"threads_started":1}
I20260925 15:11:24.442866 21965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:24.442163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:56381)
0925 15:11:24.442638 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:24.442718 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":255,"threads_started":1}
I20260925 15:11:24.870385 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:24.869681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:34685)
0925 15:11:24.870140 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:24.870223 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260925 15:11:25.003629 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:25.002849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:57597)
0925 15:11:25.003375 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:25.003472 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":320,"threads_started":1}
I20260925 15:11:25.444850 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:25.444109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:56079)
0925 15:11:25.444573 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:25.444679 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":234,"threads_started":1}
I20260925 15:11:25.871803 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:25.871184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:45557)
0925 15:11:25.871578 (+ 394us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:25.871657 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":197,"threads_started":1}
I20260925 15:11:26.005231 21970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:26.004460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:38567)
0925 15:11:26.004951 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:26.005027 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":208,"threads_started":1}
I20260925 15:11:26.446645 21971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:26.445820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:41075)
0925 15:11:26.446273 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:26.446389 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":205,"threads_started":1}
I20260925 15:11:26.873484 21972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:26.872768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:54709)
0925 15:11:26.873200 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:26.873288 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":203,"threads_started":1}
I20260925 15:11:27.006834 21973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:27.006227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:57245)
0925 15:11:27.006608 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:27.006688 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":210,"threads_started":1}
I20260925 15:11:27.448154 21974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:27.447507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:45831)
0925 15:11:27.447923 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:27.447998 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":218,"threads_started":1}
I20260925 15:11:27.875064 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:27.874393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:54919)
0925 15:11:27.874844 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:27.874918 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":226,"threads_started":1}
I20260925 15:11:28.008450 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:28.007710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:53609)
0925 15:11:28.008212 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:28.008287 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":258,"threads_started":1}
I20260925 15:11:28.449867 21977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:28.449105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:57435)
0925 15:11:28.449550 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:28.449688 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":226,"threads_started":1}
I20260925 15:11:28.876590 21978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:28.875887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:45891)
0925 15:11:28.876329 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:28.876432 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":210,"threads_started":1}
I20260925 15:11:29.010169 21979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:29.009412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:43317)
0925 15:11:29.009930 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:29.010011 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":322,"threads_started":1}
I20260925 15:11:29.451519 21980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:29.450843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:53053)
0925 15:11:29.451265 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:29.451361 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20260925 15:11:29.878168 21981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:29.877449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:51891)
0925 15:11:29.877911 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:29.877982 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":289,"threads_started":1}
I20260925 15:11:30.011700 21982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:30.011002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:45833)
0925 15:11:30.011422 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:30.011507 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":200,"threads_started":1}
I20260925 15:11:30.452920 21983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:30.452290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:56819)
0925 15:11:30.452679 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:30.452752 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":189,"threads_started":1}
I20260925 15:11:30.879611 21984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:30.878942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:47435)
0925 15:11:30.879360 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:30.879443 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":199,"threads_started":1}
I20260925 15:11:31.013291 21985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:31.012594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:47855)
0925 15:11:31.013056 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:31.013136 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":258,"threads_started":1}
I20260925 15:11:31.454485 21986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:31.453831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:54249)
0925 15:11:31.454250 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:31.454330 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":218,"threads_started":1}
I20260925 15:11:31.881047 21987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:31.880371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:52747)
0925 15:11:31.880823 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:31.880899 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":223,"threads_started":1}
I20260925 15:11:32.014876 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:32.014124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:54507)
0925 15:11:32.014587 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:32.014698 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":236,"threads_started":1}
I20260925 15:11:32.456131 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:32.455348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:37797)
0925 15:11:32.455870 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:32.455975 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":288,"threads_started":1}
I20260925 15:11:32.882557 21990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:32.881866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:55713)
0925 15:11:32.882303 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:32.882389 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":196,"threads_started":1}
I20260925 15:11:33.016273 21991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:33.015629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:33309)
0925 15:11:33.016031 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:33.016102 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":200,"threads_started":1}
I20260925 15:11:33.457767 21992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:33.456998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:41583)
0925 15:11:33.457447 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:33.457533 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20260925 15:11:33.884164 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:33.883499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:33119)
0925 15:11:33.883912 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:33.883996 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260925 15:11:34.018139 21994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:34.017446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.193:35913)
0925 15:11:34.017878 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:34.017959 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":229,"threads_started":1}
I20260925 15:11:34.459437 21995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:34.458744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.194:52525)
0925 15:11:34.459163 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:34.459267 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":230,"threads_started":1}
I20260925 15:11:34.867347 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 21518
I20260925 15:11:34.885928 21997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:34.885121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44489 (local address 127.4.205.195:58011)
0925 15:11:34.885656 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:34.885754 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.254:44489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":279,"threads_started":1}
I20260925 15:11:34.897235 21996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:34.868419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.4.205.193:37887)
0925 15:11:34.869032 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:34.869051 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:34.869100 (+ 49us) server_negotiation.cc:407] Connection header received
0925 15:11:34.870913 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:34.870922 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:34.871347 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:34.871627 (+ 280us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:34.872947 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:34.874228 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:34.875410 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:34.876022 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:34.876153 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:34.891881 (+ 15728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:34.891890 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:34.891907 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:34.891944 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:34.894639 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:34.895242 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:34.895247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:34.895253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:34.895397 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:34.895681 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:34.895684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:34.895686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:34.896152 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:34.896358 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:34.896659 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:11:34.896948 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":244,"threads_started":1}
I20260925 15:11:34.898583 21681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:34.983448 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 21518
I20260925 15:11:35.000409 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 21689
I20260925 15:11:35.016678 21999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:35.001393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.4.205.194:46881)
0925 15:11:35.001927 (+ 534us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:35.001940 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:35.001976 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:11:35.003423 (+ 1447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:35.003432 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:35.003853 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:35.004138 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:35.005386 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:35.006660 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:35.007766 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:35.008402 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:35.008525 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:35.011203 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:35.011210 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:35.011223 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:35.011261 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:35.013941 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:35.014527 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:35.014532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:35.014538 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:35.014702 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:35.014997 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:35.014999 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:35.015001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:35.015448 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:35.015697 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:35.015996 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 15:11:35.016377 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260925 15:11:35.018062 21812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:35.097797 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 21689
I20260925 15:11:35.115209 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 21820
I20260925 15:11:35.131492 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:35.116040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.4.205.195:34053)
0925 15:11:35.116587 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:35.116600 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:35.116638 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:11:35.118162 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:35.118171 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:35.118655 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:35.118929 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:35.120179 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:35.121360 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:35.122634 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:35.123255 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:35.123386 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:35.125813 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:35.125819 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:35.125832 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:35.125861 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:35.128278 (+ 2417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:35.128849 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:35.128855 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:35.128860 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:35.129051 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:35.129401 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:35.129407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:35.129415 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:35.130097 (+ 682us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:35.130412 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:35.130821 (+ 409us) server_negotiation.cc:299] Negotiation successful
0925 15:11:35.131203 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":255,"threads_started":1}
I20260925 15:11:35.132678 21943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:35.212168 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 21820
2026-09-25T15:11:35Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64420 ms)
[----------] 30 tests from SecurityITest (286267 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:35 dist-test-slave-64fx krb5kdc[22005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:35 dist-test-slave-64fx krb5kdc[22005](info): set up 2 sockets
Sep 25 15:11:35 dist-test-slave-64fx krb5kdc[22005](info): commencing operation
krb5kdc: starting...
W20260925 15:11:36.269714 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:11:36 dist-test-slave-64fx krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:11:36.487074 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41731
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41731
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:36.785223 22018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:36.785548 22018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:36.785715 22018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:36.791843 22018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:36.791952 22018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:36.792048 22018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:36.792127 22018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:36.792207 22018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:36.800843 22018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:36.802016 22018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:36.803576 22018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:36.814006 22024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:36.814037 22023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:36.814006 22026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:36.814605 22018 server_base.cc:1034] running on GCE node
I20260925 15:11:36.815276 22018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:36.816519 22018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:36.817713 22018 hybrid_clock.cc:648] HybridClock initialized: now 1790349096817681 us; error 41 us; skew 500 ppm
I20260925 15:11:36.818009 22018 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:36 dist-test-slave-64fx krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:36.820796 22018 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:11:36.822141 22018 webserver.cc:460] Webserver started at http://127.4.205.254:43291/ using document root <none> and password file <none>
I20260925 15:11:36.822779 22018 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:36.822904 22018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:36.823186 22018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:36.825887 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "c71bd1a2651d4b08874e351f3ca07e1e"
format_stamp: "Formatted at 2026-09-25 15:11:36 on dist-test-slave-64fx"
server_key: "6138f5d111f5b77fb083e0837e57253c"
server_key_iv: "774d6d8ec3ce4697cc8c1b7821c633e8"
server_key_version: "encryptionkey@0"
I20260925 15:11:36.826599 22018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "c71bd1a2651d4b08874e351f3ca07e1e"
format_stamp: "Formatted at 2026-09-25 15:11:36 on dist-test-slave-64fx"
server_key: "6138f5d111f5b77fb083e0837e57253c"
server_key_iv: "774d6d8ec3ce4697cc8c1b7821c633e8"
server_key_version: "encryptionkey@0"
I20260925 15:11:36.830919 22018 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:11:36.834025 22033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:36.835088 22018 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:11:36.835249 22018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "c71bd1a2651d4b08874e351f3ca07e1e"
format_stamp: "Formatted at 2026-09-25 15:11:36 on dist-test-slave-64fx"
server_key: "6138f5d111f5b77fb083e0837e57253c"
server_key_iv: "774d6d8ec3ce4697cc8c1b7821c633e8"
server_key_version: "encryptionkey@0"
I20260925 15:11:36.835413 22018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:36.852643 22018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:36.856019 22018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:36.856271 22018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:36.872793 22018 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41731
I20260925 15:11:36.872814 22084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41731 every 8 connection(s)
I20260925 15:11:36.873999 22018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:36.877089 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22018
I20260925 15:11:36.877312 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:36.877759 22085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:36.878136 4919 external_mini_cluster.cc:1428] Setting key 4b12dffb3bdf9d559aa9caa9547d0f16
I20260925 15:11:36.889330 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: Bootstrap starting.
Sep 25 15:11:36 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:36.892283 22085 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:36.893296 22085 log.cc:824] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:36.895876 22085 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: No bootstrap required, opened a new log
I20260925 15:11:36.898962 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:36.879984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36000 (local address 127.4.205.254:41731)
0925 15:11:36.880829 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:36.880839 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:36.880877 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:11:36.882461 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:36.882481 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:36.882847 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:36.883238 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:36.884698 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:36.885953 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:36.887207 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:36.887839 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:36.888011 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:36.892154 (+ 4143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:36.892160 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:36.892172 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:36.892201 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:36.895313 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:36.895998 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:36.896011 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:36.896017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:36.896171 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:36.896566 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:36.896572 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:36.896580 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:36.897003 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:36.897335 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:36.897762 (+ 427us) server_negotiation.cc:299] Negotiation successful
0925 15:11:36.898112 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":397,"threads_started":1}
I20260925 15:11:36.903946 22085 raft_consensus.cc:348] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } }
I20260925 15:11:36.904242 22085 raft_consensus.cc:374] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:36.904330 22085 raft_consensus.cc:729] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c71bd1a2651d4b08874e351f3ca07e1e, State: Initialized, Role: FOLLOWER
I20260925 15:11:36.904800 22085 consensus_queue.cc:260] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } }
I20260925 15:11:36.905019 22085 raft_consensus.cc:388] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:36.905160 22085 raft_consensus.cc:482] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:36.905346 22085 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:36.907310 22085 raft_consensus.cc:504] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } }
I20260925 15:11:36.907742 22085 leader_election.cc:302] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c71bd1a2651d4b08874e351f3ca07e1e; no voters:
I20260925 15:11:36.908234 22085 leader_election.cc:288] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:36.908340 22090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:36.908910 22090 raft_consensus.cc:686] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [term 1 LEADER]: Becoming Leader. State: Replica: c71bd1a2651d4b08874e351f3ca07e1e, State: Running, Role: LEADER
I20260925 15:11:36.909430 22090 consensus_queue.cc:237] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } }
I20260925 15:11:36.909982 22085 sys_catalog.cc:564] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:36.912336 22092 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c71bd1a2651d4b08874e351f3ca07e1e. Latest consensus state: current_term: 1 leader_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } } }
I20260925 15:11:36.912314 22091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c71bd1a2651d4b08874e351f3ca07e1e" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41731 } } }
I20260925 15:11:36.912688 22091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:36.912688 22092 sys_catalog.cc:458] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:36.915138 22096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:36.919066 22096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:36.926574 22096 catalog_manager.cc:1347] Generated new cluster ID: 8faa42ad718145659d9b2fac8bd1cf39
I20260925 15:11:36.926671 22096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:36.937916 22108 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:11:36.951429 22096 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:36.952472 22096 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:36.969256 22096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: Generated new TSK 0
I20260925 15:11:36.970019 22096 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:11:37.029731 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:37.340835 22111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:37.341199 22111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:37.341360 22111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:37.348294 22111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:37.348423 22111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:37.348601 22111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:37.357470 22111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:37.358544 22111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:37.359694 22111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:37.371412 22116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:37.371376 22117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:37.371345 22119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:37.372399 22111 server_base.cc:1034] running on GCE node
I20260925 15:11:37.372916 22111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:37.373771 22111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:37.374949 22111 hybrid_clock.cc:648] HybridClock initialized: now 1790349097374917 us; error 29 us; skew 500 ppm
I20260925 15:11:37.375212 22111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:37 dist-test-slave-64fx krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:37.378018 22111 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:11:37.379307 22111 webserver.cc:460] Webserver started at http://127.4.205.193:37929/ using document root <none> and password file <none>
I20260925 15:11:37.379886 22111 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:37.380017 22111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:37.380306 22111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:37.383003 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "593734645f954799b97894b0603d0ad2"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "873284caae85a683922a26c84d56a1f4"
server_key_iv: "7988408ab86f7effd43238687e4fef3c"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.383733 22111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "593734645f954799b97894b0603d0ad2"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "873284caae85a683922a26c84d56a1f4"
server_key_iv: "7988408ab86f7effd43238687e4fef3c"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.388208 22111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:37.391314 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.392428 22111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:11:37.392582 22111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "593734645f954799b97894b0603d0ad2"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "873284caae85a683922a26c84d56a1f4"
server_key_iv: "7988408ab86f7effd43238687e4fef3c"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.392752 22111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:37.421813 22111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:37.425271 22111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:37.425513 22111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:37.426364 22111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:37.427584 22111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:37.427676 22111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.427821 22111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:37.427883 22111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.458330 22111 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:42349
I20260925 15:11:37.458402 22238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:42349 every 8 connection(s)
I20260925 15:11:37.459548 22111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:37.461663 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22111
I20260925 15:11:37.461867 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:37.462673 4919 external_mini_cluster.cc:1428] Setting key ad18aee084af8ca9b8000ce2677c8bde
Sep 25 15:11:37 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:37.480505 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:37.462228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:59031 (local address 127.4.205.254:41731)
0925 15:11:37.462692 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:37.462697 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:37.463919 (+ 1222us) server_negotiation.cc:407] Connection header received
0925 15:11:37.466007 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:37.466015 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:37.466098 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:37.466208 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:37.467640 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.468793 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.470545 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.471446 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.471601 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:37.474937 (+ 3336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:37.474942 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:37.474945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:37.474976 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:37.477546 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:37.478265 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:37.478273 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:37.478277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:37.478413 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:37.478851 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:37.478854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:37.478856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:37.479102 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:37.479247 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:37.479925 (+ 678us) server_negotiation.cc:299] Negotiation successful
0925 15:11:37.480140 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":234,"threads_started":1}
I20260925 15:11:37.481410 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:37.462943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41731 (local address 127.4.205.193:59031)
0925 15:11:37.463740 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:37.463807 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:37.465765 (+ 1958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:37.466393 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:37.466401 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:37.466869 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:37.467468 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:37.467486 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.468978 (+ 1492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.468985 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:37.470403 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:37.470414 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.471657 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.471662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:37.471839 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:37.472497 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:37.472516 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:37.474725 (+ 2209us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:37.477743 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:37.477752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:37.477768 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:37.478145 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:37.478527 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:37.478533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:37.478537 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:37.478731 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:37.479241 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:37.479248 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:37.479556 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 15:11:37.479804 (+ 248us) client_negotiation.cc:240] Negotiation successful
0925 15:11:37.480333 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":354,"threads_started":1}
I20260925 15:11:37.483065 22239 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41731
I20260925 15:11:37.483351 22239 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:37.484009 22239 heartbeater.cc:511] Master 127.4.205.254:41731 requested a full tablet report, sending...
I20260925 15:11:37.486466 22050 ts_manager.cc:194] Registered new tserver with Master: 593734645f954799b97894b0603d0ad2 (127.4.205.193:42349)
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:11:37.488266 22050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:59031
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:11:37.514734 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:37.812642 22245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:37.812862 22245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:37.813023 22245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:37.819649 22245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:37.819727 22245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:37.819856 22245 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:37.828893 22245 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:37.830045 22245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:37.831209 22245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:37.843667 22253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:37.843703 22251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:37.844113 22250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:37.844545 22245 server_base.cc:1034] running on GCE node
I20260925 15:11:37.845108 22245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:37.846005 22245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:37.847209 22245 hybrid_clock.cc:648] HybridClock initialized: now 1790349097847160 us; error 56 us; skew 500 ppm
I20260925 15:11:37.847499 22245 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:37 dist-test-slave-64fx krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:37.850323 22245 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:11:37.851543 22245 webserver.cc:460] Webserver started at http://127.4.205.194:43245/ using document root <none> and password file <none>
I20260925 15:11:37.852095 22245 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:37.852185 22245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:37.852416 22245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:37.855050 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "cfd83a94683f47b88a18f88d6d56d54f"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "1425b852ba2efac5e9b840b58e6f69e1"
server_key_iv: "53120e00cb3ea985a3e5002cacb9969a"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.855708 22245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "cfd83a94683f47b88a18f88d6d56d54f"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "1425b852ba2efac5e9b840b58e6f69e1"
server_key_iv: "53120e00cb3ea985a3e5002cacb9969a"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.860204 22245 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:11:37.863266 22260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.864359 22245 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:11:37.864473 22245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "cfd83a94683f47b88a18f88d6d56d54f"
format_stamp: "Formatted at 2026-09-25 15:11:37 on dist-test-slave-64fx"
server_key: "1425b852ba2efac5e9b840b58e6f69e1"
server_key_iv: "53120e00cb3ea985a3e5002cacb9969a"
server_key_version: "encryptionkey@0"
I20260925 15:11:37.864619 22245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:37.876431 22245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:37.880204 22245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:37.880470 22245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:37.881346 22245 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:37.882587 22245 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:37.882654 22245 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.882810 22245 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:37.882866 22245 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:37.913429 22245 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:38489
I20260925 15:11:37.913470 22372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:38489 every 8 connection(s)
I20260925 15:11:37.914660 22245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:37.924059 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22245
I20260925 15:11:37.924244 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:37.924633 4919 external_mini_cluster.cc:1428] Setting key 3e0f92789004d0efc3926a9fa44543cb
Sep 25 15:11:37 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349097, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:37.933307 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:37.916917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:34127 (local address 127.4.205.254:41731)
0925 15:11:37.917091 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:37.917095 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:37.918039 (+ 944us) server_negotiation.cc:407] Connection header received
0925 15:11:37.919777 (+ 1738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:37.919783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:37.919861 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:37.919988 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:37.921414 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.922278 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.923870 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.924347 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.924461 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:37.927912 (+ 3451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:37.927917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:37.927920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:37.927949 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:37.930508 (+ 2559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:37.931189 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:37.931196 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:37.931200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:37.931326 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:37.931735 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:37.931741 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:37.931745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:37.931993 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:37.932167 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:37.932763 (+ 596us) server_negotiation.cc:299] Negotiation successful
0925 15:11:37.932975 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:11:37.934255 22375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:37.917254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41731 (local address 127.4.205.194:34127)
0925 15:11:37.917904 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:37.917937 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:37.919543 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:37.920218 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:37.920226 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:37.920701 (+ 475us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:37.921249 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:37.921265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.922408 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.922417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:37.923719 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:37.923729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:37.924553 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:37.924560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:37.924825 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:37.925448 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:37.925468 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:37.927720 (+ 2252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:37.930646 (+ 2926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:37.930655 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:37.930669 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:37.931061 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:37.931426 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:37.931432 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:37.931437 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:37.931628 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:37.932132 (+ 504us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:37.932139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:37.932431 (+ 292us) client_negotiation.cc:769] Sending connection context
0925 15:11:37.932668 (+ 237us) client_negotiation.cc:240] Negotiation successful
0925 15:11:37.933152 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":280,"threads_started":1}
I20260925 15:11:37.935653 22373 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41731
I20260925 15:11:37.935957 22373 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:37.936599 22373 heartbeater.cc:511] Master 127.4.205.254:41731 requested a full tablet report, sending...
I20260925 15:11:37.938046 22050 ts_manager.cc:194] Registered new tserver with Master: cfd83a94683f47b88a18f88d6d56d54f (127.4.205.194:38489)
I20260925 15:11:37.939275 22050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:34127
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:11:37.972411 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:38.276865 22378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:38.277153 22378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:38.277247 22378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:38.284749 22378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:38.284868 22378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:38.284981 22378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:38.294967 22378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:38.296238 22378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:38.297523 22378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:38.309864 22386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:38.309944 22383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:38.309919 22384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:38.310845 22378 server_base.cc:1034] running on GCE node
I20260925 15:11:38.311475 22378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:38.312278 22378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:38.313495 22378 hybrid_clock.cc:648] HybridClock initialized: now 1790349098313448 us; error 23 us; skew 500 ppm
I20260925 15:11:38.313848 22378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:38 dist-test-slave-64fx krb5kdc[22005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:38.316720 22378 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:11:38.318172 22378 webserver.cc:460] Webserver started at http://127.4.205.195:45933/ using document root <none> and password file <none>
I20260925 15:11:38.318840 22378 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:38.318977 22378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:38.319276 22378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:38.322157 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "537d71a2b09241ec85789ef1b777ca4a"
format_stamp: "Formatted at 2026-09-25 15:11:38 on dist-test-slave-64fx"
server_key: "9a2b708c32b2c3ed35c8c7f20d279c65"
server_key_iv: "8aeee1f50074ab7230c55cecc6ab7f95"
server_key_version: "encryptionkey@0"
I20260925 15:11:38.322893 22378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "537d71a2b09241ec85789ef1b777ca4a"
format_stamp: "Formatted at 2026-09-25 15:11:38 on dist-test-slave-64fx"
server_key: "9a2b708c32b2c3ed35c8c7f20d279c65"
server_key_iv: "8aeee1f50074ab7230c55cecc6ab7f95"
server_key_version: "encryptionkey@0"
I20260925 15:11:38.327608 22378 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:38.330834 22393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:38.332077 22378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:11:38.332237 22378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "537d71a2b09241ec85789ef1b777ca4a"
format_stamp: "Formatted at 2026-09-25 15:11:38 on dist-test-slave-64fx"
server_key: "9a2b708c32b2c3ed35c8c7f20d279c65"
server_key_iv: "8aeee1f50074ab7230c55cecc6ab7f95"
server_key_version: "encryptionkey@0"
I20260925 15:11:38.332424 22378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:38.365653 22378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:38.369804 22378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:38.370103 22378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:38.371080 22378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:38.372382 22378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:38.372478 22378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:38.372622 22378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:38.372689 22378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:38.408430 22378 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:39669
I20260925 15:11:38.408504 22505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:39669 every 8 connection(s)
I20260925 15:11:38.409684 22378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:38.412596 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22378
I20260925 15:11:38.412783 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:38.413379 4919 external_mini_cluster.cc:1428] Setting key b0015aa61898e9c71fe2edd8270db64f
Sep 25 15:11:38 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349098, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:38.428555 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.412196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:46389 (local address 127.4.205.254:41731)
0925 15:11:38.412418 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.412427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.413822 (+ 1395us) server_negotiation.cc:407] Connection header received
0925 15:11:38.416207 (+ 2385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.416212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.416300 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.416377 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:38.417839 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.418533 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.419902 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.420296 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.420382 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.423490 (+ 3108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:38.423494 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:38.423497 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:38.423528 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:38.425790 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.426400 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.426404 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.426406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.426509 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.426837 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.426840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.426842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.427085 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:38.427240 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.427992 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.428230 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:11:38.429425 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.412854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41731 (local address 127.4.205.195:46389)
0925 15:11:38.413679 (+ 825us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.413721 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.415939 (+ 2218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.416604 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.416615 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.417118 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:38.417695 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.417712 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.418649 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.418652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.419794 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.419804 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.420461 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.420466 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.420646 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.421202 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:38.421226 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:38.423333 (+ 2107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:38.425892 (+ 2559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:38.425898 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:38.425908 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:38.426288 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:38.426587 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:38.426590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:38.426592 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:38.426751 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:38.427244 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:38.427253 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:38.427629 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.427907 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.428360 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":329,"threads_started":1}
I20260925 15:11:38.430590 22506 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41731
I20260925 15:11:38.430831 22506 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:38.431424 22506 heartbeater.cc:511] Master 127.4.205.254:41731 requested a full tablet report, sending...
I20260925 15:11:38.432583 22050 ts_manager.cc:194] Registered new tserver with Master: 537d71a2b09241ec85789ef1b777ca4a (127.4.205.195:39669)
I20260925 15:11:38.433337 22050 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:46389
I20260925 15:11:38.440358 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:38.455339 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.444456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.4.205.254:41731)
0925 15:11:38.444588 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.444593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.445056 (+ 463us) server_negotiation.cc:407] Connection header received
0925 15:11:38.445179 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.445182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.445234 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.445306 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:38.446567 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.447185 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.448462 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.448812 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.448889 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.451033 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:38.451038 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:38.451041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:38.451069 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:38.453254 (+ 2185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.453896 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.453900 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.453902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.454021 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.454312 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.454315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.454317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.454574 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:38.454752 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.455003 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.455141 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:11:38.458968 22050 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:38.465399 22050 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:38.491158 22239 heartbeater.cc:503] Master 127.4.205.254:41731 was elected leader, sending a full tablet report...
I20260925 15:11:38.497735 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.476317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.4.205.193:42349)
0925 15:11:38.488884 (+ 12567us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.488893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.488918 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:11:38.489670 (+ 752us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.489682 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.489938 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.490163 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:38.491034 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.492912 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.494845 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.496407 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.496542 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.496841 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.497203 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.497387 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12365,"thread_start_us":12927,"threads_started":1}
I20260925 15:11:38.497792 22518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.476165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:42349 (local address 127.0.0.1:43980)
0925 15:11:38.488455 (+ 12290us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.488509 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.488688 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.490449 (+ 1761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.490458 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.490485 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:38.490868 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.490879 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.493090 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.493097 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.494713 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.494724 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.496620 (+ 1896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.496626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.496940 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.497001 (+ 61us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.497154 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.497354 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12043,"mutex_wait_us":24,"thread_start_us":12680,"threads_started":1}
I20260925 15:11:38.498653 22521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.476591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:38489 (local address 127.0.0.1:49124)
0925 15:11:38.490561 (+ 13970us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.490584 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.490689 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.491591 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.491598 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.491625 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:38.492010 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.492023 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.493849 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.493854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.495896 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.495914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.497944 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.497951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.498066 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.498114 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.498217 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.498336 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13822,"thread_start_us":13692,"threads_started":1}
I20260925 15:11:38.498735 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.477041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49124 (local address 127.4.205.194:38489)
0925 15:11:38.490106 (+ 13065us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.490114 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.490820 (+ 706us) server_negotiation.cc:407] Connection header received
0925 15:11:38.490921 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.490931 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.491194 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.491374 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:38.492212 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.493704 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.496082 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.497457 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.497608 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.497796 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.498264 (+ 468us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.498420 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12881,"thread_start_us":12755,"threads_started":1}
I20260925 15:11:38.500515 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.476856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34554 (local address 127.4.205.195:39669)
0925 15:11:38.491040 (+ 14184us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.491049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.491079 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:11:38.491184 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.491192 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.491407 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.493190 (+ 1783us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:38.493389 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.494778 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.497402 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.499050 (+ 1648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.499922 (+ 872us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.500121 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.500172 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.500276 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13990,"thread_start_us":13817,"threads_started":1}
I20260925 15:11:38.501003 22519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.476265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:39669 (local address 127.0.0.1:34554)
0925 15:11:38.488455 (+ 12190us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.489441 (+ 986us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.489547 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.492686 (+ 3139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.492690 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.492717 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:38.493045 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.493053 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.494948 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.494953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.497163 (+ 2210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.497177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.499257 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.499263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.499432 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.499459 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.499520 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.499644 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12003,"mutex_wait_us":23,"thread_start_us":11902,"threads_started":1}
I20260925 15:11:38.502019 22308 tablet_service.cc:1467] Processing CreateTablet for tablet a89709b5da8b42cc9debc6854357fd2c (DEFAULT_TABLE table=test-table [id=0b540ef724ba4dff91eeec872972d031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:38.501881 22174 tablet_service.cc:1467] Processing CreateTablet for tablet a89709b5da8b42cc9debc6854357fd2c (DEFAULT_TABLE table=test-table [id=0b540ef724ba4dff91eeec872972d031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:38.503309 22308 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89709b5da8b42cc9debc6854357fd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:38.503304 22174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89709b5da8b42cc9debc6854357fd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:38.503835 22441 tablet_service.cc:1467] Processing CreateTablet for tablet a89709b5da8b42cc9debc6854357fd2c (DEFAULT_TABLE table=test-table [id=0b540ef724ba4dff91eeec872972d031]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:38.505168 22441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a89709b5da8b42cc9debc6854357fd2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:38.518661 22526 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Bootstrap starting.
I20260925 15:11:38.518740 22524 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Bootstrap starting.
I20260925 15:11:38.521730 22526 tablet_bootstrap.cc:654] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:38.522256 22525 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Bootstrap starting.
I20260925 15:11:38.522970 22524 tablet_bootstrap.cc:654] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:38.523016 22526 log.cc:824] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:38.524147 22524 log.cc:824] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:38.525354 22525 tablet_bootstrap.cc:654] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:38.525820 22526 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: No bootstrap required, opened a new log
I20260925 15:11:38.526069 22526 ts_tablet_manager.cc:1397] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260925 15:11:38.526429 22525 log.cc:824] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:38.526983 22524 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: No bootstrap required, opened a new log
I20260925 15:11:38.527292 22524 ts_tablet_manager.cc:1397] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 15:11:38.528879 22525 tablet_bootstrap.cc:492] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: No bootstrap required, opened a new log
I20260925 15:11:38.529098 22525 ts_tablet_manager.cc:1397] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:11:38.535552 22525 raft_consensus.cc:348] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.535890 22525 raft_consensus.cc:374] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:38.536056 22525 raft_consensus.cc:729] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfd83a94683f47b88a18f88d6d56d54f, State: Initialized, Role: FOLLOWER
I20260925 15:11:38.536192 22526 raft_consensus.cc:348] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.536656 22526 raft_consensus.cc:374] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:38.536775 22526 raft_consensus.cc:729] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 593734645f954799b97894b0603d0ad2, State: Initialized, Role: FOLLOWER
I20260925 15:11:38.536643 22525 consensus_queue.cc:260] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.537330 22526 consensus_queue.cc:260] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.537387 22524 raft_consensus.cc:348] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.537781 22524 raft_consensus.cc:374] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:38.537937 22524 raft_consensus.cc:729] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 537d71a2b09241ec85789ef1b777ca4a, State: Initialized, Role: FOLLOWER
I20260925 15:11:38.538614 22524 consensus_queue.cc:260] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.539237 22526 ts_tablet_manager.cc:1428] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Time spent starting tablet: real 0.013s user 0.012s sys 0.001s
I20260925 15:11:38.542467 22506 heartbeater.cc:503] Master 127.4.205.254:41731 was elected leader, sending a full tablet report...
I20260925 15:11:38.542932 22524 ts_tablet_manager.cc:1428] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260925 15:11:38.543052 22373 heartbeater.cc:503] Master 127.4.205.254:41731 was elected leader, sending a full tablet report...
I20260925 15:11:38.544085 22525 ts_tablet_manager.cc:1428] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Time spent starting tablet: real 0.015s user 0.009s sys 0.005s
W20260925 15:11:38.661350 22507 tablet.cc:2369] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:11:38.666549 22374 tablet.cc:2369] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:11:38.711844 22240 tablet.cc:2369] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:38.762486 22532 raft_consensus.cc:482] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:38.762786 22532 raft_consensus.cc:504] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.764534 22532 leader_election.cc:288] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 593734645f954799b97894b0603d0ad2 (127.4.205.193:42349), cfd83a94683f47b88a18f88d6d56d54f (127.4.205.194:38489)
I20260925 15:11:38.771090 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.764886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:49047 (local address 127.4.205.193:42349)
0925 15:11:38.765094 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.765100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.765120 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:11:38.765262 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.765266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.765322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.765436 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:38.766067 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.767117 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.769181 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.770327 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.770441 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.770493 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.770828 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.770915 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:11:38.771407 22508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.764773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:42349 (local address 127.4.205.195:49047)
0925 15:11:38.764941 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.764964 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.765105 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.765492 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.765501 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.765545 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:38.765896 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.765914 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.767226 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.767232 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.769042 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.769053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.770460 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.770466 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.770660 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.770697 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.770788 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.770917 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:11:38.772152 22533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.765152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:38489 (local address 127.4.205.195:33963)
0925 15:11:38.766017 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:38.766035 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:38.766158 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:38.766447 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:38.766451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:38.766474 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:38.766788 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.766798 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.768078 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.768082 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.769940 (+ 1858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:38.769954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.771546 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.771554 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:38.771754 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.771776 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:11:38.771855 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:11:38.771929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":536,"threads_started":1}
I20260925 15:11:38.772271 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.765293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:33963 (local address 127.4.205.194:38489)
0925 15:11:38.765509 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.765515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.766115 (+ 600us) server_negotiation.cc:407] Connection header received
0925 15:11:38.766283 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.766286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.766341 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.766406 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:38.766932 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.767951 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.770081 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.771142 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.771346 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.771416 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.771910 (+ 494us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.772039 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:11:38.772475 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89709b5da8b42cc9debc6854357fd2c" candidate_uuid: "537d71a2b09241ec85789ef1b777ca4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593734645f954799b97894b0603d0ad2" is_pre_election: true
I20260925 15:11:38.773010 22194 raft_consensus.cc:2454] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 537d71a2b09241ec85789ef1b777ca4a in term 0.
I20260925 15:11:38.773154 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89709b5da8b42cc9debc6854357fd2c" candidate_uuid: "537d71a2b09241ec85789ef1b777ca4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd83a94683f47b88a18f88d6d56d54f" is_pre_election: true
I20260925 15:11:38.773700 22397 leader_election.cc:302] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 537d71a2b09241ec85789ef1b777ca4a, 593734645f954799b97894b0603d0ad2; no voters:
I20260925 15:11:38.773700 22328 raft_consensus.cc:2454] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 537d71a2b09241ec85789ef1b777ca4a in term 0.
I20260925 15:11:38.774067 22532 raft_consensus.cc:2781] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:38.774238 22532 raft_consensus.cc:482] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:38.774372 22532 raft_consensus.cc:3037] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:38.776501 22532 raft_consensus.cc:504] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.777105 22532 leader_election.cc:288] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [CANDIDATE]: Term 1 election: Requested vote from peers 593734645f954799b97894b0603d0ad2 (127.4.205.193:42349), cfd83a94683f47b88a18f88d6d56d54f (127.4.205.194:38489)
I20260925 15:11:38.777446 22194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89709b5da8b42cc9debc6854357fd2c" candidate_uuid: "537d71a2b09241ec85789ef1b777ca4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "593734645f954799b97894b0603d0ad2"
I20260925 15:11:38.777673 22194 raft_consensus.cc:3037] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:38.777806 22328 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a89709b5da8b42cc9debc6854357fd2c" candidate_uuid: "537d71a2b09241ec85789ef1b777ca4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cfd83a94683f47b88a18f88d6d56d54f"
I20260925 15:11:38.778038 22328 raft_consensus.cc:3037] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:38.779903 22194 raft_consensus.cc:2454] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 537d71a2b09241ec85789ef1b777ca4a in term 1.
I20260925 15:11:38.780174 22328 raft_consensus.cc:2454] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 537d71a2b09241ec85789ef1b777ca4a in term 1.
I20260925 15:11:38.780269 22397 leader_election.cc:302] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 537d71a2b09241ec85789ef1b777ca4a, 593734645f954799b97894b0603d0ad2; no voters:
I20260925 15:11:38.780568 22532 raft_consensus.cc:2781] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:38.780977 22532 raft_consensus.cc:686] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 1 LEADER]: Becoming Leader. State: Replica: 537d71a2b09241ec85789ef1b777ca4a, State: Running, Role: LEADER
I20260925 15:11:38.781481 22532 consensus_queue.cc:237] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } }
I20260925 15:11:38.785678 22049 catalog_manager.cc:5495] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a reported cstate change: term changed from 0 to 1, leader changed from <none> to 537d71a2b09241ec85789ef1b777ca4a (127.4.205.195). New cstate: current_term: 1 leader_uuid: "537d71a2b09241ec85789ef1b777ca4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "537d71a2b09241ec85789ef1b777ca4a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 39669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 } health_report { overall_health: UNKNOWN } } }
I20260925 15:11:38.860040 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.855292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34564 (local address 127.4.205.195:39669)
0925 15:11:38.855447 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.855451 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.855547 (+ 96us) server_negotiation.cc:407] Connection header received
0925 15:11:38.855721 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.855724 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.855810 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.855944 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:38.856252 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.856954 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.858275 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.858662 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.858749 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.859131 (+ 382us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:38.859610 (+ 479us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:38.859718 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.859776 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.859834 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:11:38.867349 22194 raft_consensus.cc:1261] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 1 FOLLOWER]: Refusing update from remote peer 537d71a2b09241ec85789ef1b777ca4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:38.867389 22328 raft_consensus.cc:1261] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 1 FOLLOWER]: Refusing update from remote peer 537d71a2b09241ec85789ef1b777ca4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:38.868245 22534 consensus_queue.cc:1035] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "593734645f954799b97894b0603d0ad2" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 42349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:38.868577 22532 consensus_queue.cc:1035] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cfd83a94683f47b88a18f88d6d56d54f" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 38489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:38.904682 22048 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:38.905015 22048 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:38.909200 22048 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c71bd1a2651d4b08874e351f3ca07e1e: Sending DeleteTablet for 3 replicas of tablet a89709b5da8b42cc9debc6854357fd2c
I20260925 15:11:38.910171 22174 tablet_service.cc:1514] Processing DeleteTablet for tablet a89709b5da8b42cc9debc6854357fd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:38 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:43980
I20260925 15:11:38.910459 22441 tablet_service.cc:1514] Processing DeleteTablet for tablet a89709b5da8b42cc9debc6854357fd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:38 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:34554
I20260925 15:11:38.910655 22308 tablet_service.cc:1514] Processing DeleteTablet for tablet a89709b5da8b42cc9debc6854357fd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:38 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49124
I20260925 15:11:38.910712 22307 tablet_service.cc:1514] Processing DeleteTablet for tablet a89709b5da8b42cc9debc6854357fd2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:38 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:49124
I20260925 15:11:38.913471 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22111
I20260925 15:11:38.939567 22549 tablet_replica.cc:330] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: stopping tablet replica
I20260925 15:11:38.940104 22549 raft_consensus.cc:2229] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:38.940725 22549 raft_consensus.cc:2258] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:38.945914 22549 ts_tablet_manager.cc:1905] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:38.949151 22548 tablet_replica.cc:330] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: stopping tablet replica
I20260925 15:11:38.949247 22547 tablet_replica.cc:330] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: stopping tablet replica
I20260925 15:11:38.949601 22548 raft_consensus.cc:2229] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:38.949620 22547 raft_consensus.cc:2229] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:38.950115 22547 raft_consensus.cc:2258] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:38.950342 22548 raft_consensus.cc:2258] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:38.952255 22547 ts_tablet_manager.cc:1905] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:38.953884 22548 ts_tablet_manager.cc:1905] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:38.959227 22547 ts_tablet_manager.cc:1918] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:38.959373 22547 log.cc:1196] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/a89709b5da8b42cc9debc6854357fd2c
I20260925 15:11:38.959883 22547 ts_tablet_manager.cc:1939] T a89709b5da8b42cc9debc6854357fd2c P 593734645f954799b97894b0603d0ad2: Deleting consensus metadata
I20260925 15:11:38.960367 22549 ts_tablet_manager.cc:1918] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:38.960556 22549 log.cc:1196] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/a89709b5da8b42cc9debc6854357fd2c
I20260925 15:11:38.960987 22548 ts_tablet_manager.cc:1918] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:38.961158 22548 log.cc:1196] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/a89709b5da8b42cc9debc6854357fd2c
I20260925 15:11:38.961165 22549 ts_tablet_manager.cc:1939] T a89709b5da8b42cc9debc6854357fd2c P cfd83a94683f47b88a18f88d6d56d54f: Deleting consensus metadata
I20260925 15:11:38.961740 22548 ts_tablet_manager.cc:1939] T a89709b5da8b42cc9debc6854357fd2c P 537d71a2b09241ec85789ef1b777ca4a: Deleting consensus metadata
I20260925 15:11:38.961732 22037 catalog_manager.cc:4841] TS 593734645f954799b97894b0603d0ad2 (127.4.205.193:42349): tablet a89709b5da8b42cc9debc6854357fd2c (table test-table [id=0b540ef724ba4dff91eeec872972d031]) successfully deleted
I20260925 15:11:38.963490 22034 catalog_manager.cc:4841] TS 537d71a2b09241ec85789ef1b777ca4a (127.4.205.195:39669): tablet a89709b5da8b42cc9debc6854357fd2c (table test-table [id=0b540ef724ba4dff91eeec872972d031]) successfully deleted
Sep 25 15:11:38 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:11:38.964888 22036 catalog_manager.cc:4841] TS cfd83a94683f47b88a18f88d6d56d54f (127.4.205.194:38489): tablet a89709b5da8b42cc9debc6854357fd2c (table test-table [id=0b540ef724ba4dff91eeec872972d031]) successfully deleted
W20260925 15:11:38.965287 22036 catalog_manager.cc:4808] TS cfd83a94683f47b88a18f88d6d56d54f (127.4.205.194:38489): delete failed for tablet a89709b5da8b42cc9debc6854357fd2c because the tablet was not found. No further retry: Not found: Tablet not found: a89709b5da8b42cc9debc6854357fd2c
I20260925 15:11:38.971096 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:38.915556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.4.205.193:42349)
0925 15:11:38.915763 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:38.915771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:38.939385 (+ 23614us) server_negotiation.cc:407] Connection header received
0925 15:11:38.941400 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:38.941410 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:38.941532 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:38.941645 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:38.944344 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.945916 (+ 1572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.950669 (+ 4753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:38.951349 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:38.951450 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:38.965677 (+ 14227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:38.965688 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:38.965699 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:38.965733 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:38.968398 (+ 2665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.969057 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.969063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.969066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.969189 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:38.969582 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:38.969588 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:38.969592 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:38.969902 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:38.970226 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:38.970541 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:11:38.970772 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:11:38.971858 22234 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:39.061007 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22111
I20260925 15:11:39.086015 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22245
Sep 25 15:11:39 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:11:39.102600 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:39.086782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.4.205.194:38489)
0925 15:11:39.086961 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:39.086966 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:39.086988 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:11:39.087162 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:39.087165 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:39.087226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:39.087311 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:39.088883 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.089611 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.091463 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.091849 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.091938 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:39.096931 (+ 4993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:39.096946 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:39.096962 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:39.096999 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:39.099794 (+ 2795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:39.100483 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:39.100491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:39.100493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:39.100629 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:39.100994 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:39.101001 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:39.101009 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:39.101386 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:39.101719 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:39.102029 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 15:11:39.102272 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:11:39.103278 22368 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:39.192063 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22245
I20260925 15:11:39.208923 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22378
Sep 25 15:11:39 dist-test-slave-64fx krb5kdc[22005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:11:39.225508 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:39.210188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.4.205.195:39669)
0925 15:11:39.210363 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:39.210367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:39.210386 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:11:39.210439 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:39.210443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:39.210540 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:39.210659 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:39.211943 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.212739 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.214263 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.214699 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.214835 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:39.219164 (+ 4329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:39.219176 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:39.219190 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:39.219233 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:39.222564 (+ 3331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:39.223303 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:39.223316 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:39.223319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:39.223520 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:39.223959 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:39.223963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:39.223965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:39.224306 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:39.224628 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:39.224923 (+ 295us) server_negotiation.cc:299] Negotiation successful
0925 15:11:39.225153 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:11:39.226325 22501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:39.350967 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22378
I20260925 15:11:39.370294 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22018
I20260925 15:11:39.371420 22080 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:39.497975 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22018
2026-09-25T15:11:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4289 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:11:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:39Z Disabled control of system clock
I20260925 15:11:39.569751 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:46663
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:46663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:39.878752 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:39.879012 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:39.879096 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:39.886097 22558 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:39.886194 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:39.886284 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:39.886386 22558 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:39.886464 22558 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:39.895002 22558 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:46663
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:46663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:39.896382 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:39.898275 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:39.909767 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:39.909813 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:39.909834 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:39.910605 22558 server_base.cc:1034] running on GCE node
I20260925 15:11:39.911279 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:39.912551 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:39.913728 22558 hybrid_clock.cc:648] HybridClock initialized: now 1790349099913670 us; error 51 us; skew 500 ppm
I20260925 15:11:39.914062 22558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:39.919314 22558 webserver.cc:460] Webserver started at http://127.4.205.254:40857/ using document root <none> and password file <none>
I20260925 15:11:39.919963 22558 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:39.920070 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:39.920436 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:39.923280 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a"
format_stamp: "Formatted at 2026-09-25 15:11:39 on dist-test-slave-64fx"
server_key: "ae39a649c3406dbd2fab38adbbf15451"
server_key_iv: "e663cc1926a3564155946b882c881522"
server_key_version: "encryptionkey@0"
I20260925 15:11:39.924022 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a"
format_stamp: "Formatted at 2026-09-25 15:11:39 on dist-test-slave-64fx"
server_key: "ae39a649c3406dbd2fab38adbbf15451"
server_key_iv: "e663cc1926a3564155946b882c881522"
server_key_version: "encryptionkey@0"
I20260925 15:11:39.928685 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:11:39.931878 22572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:39.933122 22558 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:11:39.933295 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a"
format_stamp: "Formatted at 2026-09-25 15:11:39 on dist-test-slave-64fx"
server_key: "ae39a649c3406dbd2fab38adbbf15451"
server_key_iv: "e663cc1926a3564155946b882c881522"
server_key_version: "encryptionkey@0"
I20260925 15:11:39.933488 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:39.953858 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:39.954586 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:39.954850 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:39.972805 22558 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:46663
I20260925 15:11:39.972836 22623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:46663 every 8 connection(s)
I20260925 15:11:39.974174 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:39.977466 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22558
I20260925 15:11:39.977526 22624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:39.977715 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:39.978148 4919 external_mini_cluster.cc:1428] Setting key 84138c63e96a47970581128791db7e7b
I20260925 15:11:39.993594 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: Bootstrap starting.
I20260925 15:11:39.995635 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:39.979933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47528 (local address 127.4.205.254:46663)
0925 15:11:39.981479 (+ 1546us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:39.981495 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:39.981549 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:11:39.984853 (+ 3304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:39.984880 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:39.985455 (+ 575us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:39.985928 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:39.986821 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.988683 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.990670 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:39.991789 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:39.991960 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:39.993014 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:39.993025 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:39.993041 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:39.993748 (+ 707us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:39.993837 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:39.993866 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:39.993982 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:39.994157 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:11:39.994466 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":411,"threads_started":1}
I20260925 15:11:39.997344 22624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:39.998521 22624 log.cc:824] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:40.000900 22624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: No bootstrap required, opened a new log
I20260925 15:11:40.008090 22624 raft_consensus.cc:348] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } }
I20260925 15:11:40.008419 22624 raft_consensus.cc:374] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:40.008525 22624 raft_consensus.cc:729] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da9d0b9d4dab4deb990c44e3fa3fd56a, State: Initialized, Role: FOLLOWER
I20260925 15:11:40.009037 22624 consensus_queue.cc:260] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } }
I20260925 15:11:40.009266 22624 raft_consensus.cc:388] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:40.009423 22624 raft_consensus.cc:482] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:40.009586 22624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:40.011893 22624 raft_consensus.cc:504] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } }
I20260925 15:11:40.012387 22624 leader_election.cc:302] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da9d0b9d4dab4deb990c44e3fa3fd56a; no voters:
I20260925 15:11:40.012949 22624 leader_election.cc:288] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:40.013077 22629 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:40.013770 22629 raft_consensus.cc:686] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [term 1 LEADER]: Becoming Leader. State: Replica: da9d0b9d4dab4deb990c44e3fa3fd56a, State: Running, Role: LEADER
I20260925 15:11:40.014212 22629 consensus_queue.cc:237] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } }
I20260925 15:11:40.014894 22624 sys_catalog.cc:564] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:40.016688 22630 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } } }
I20260925 15:11:40.016793 22631 sys_catalog.cc:455] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [sys.catalog]: SysCatalogTable state changed. Reason: New leader da9d0b9d4dab4deb990c44e3fa3fd56a. Latest consensus state: current_term: 1 leader_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da9d0b9d4dab4deb990c44e3fa3fd56a" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 46663 } } }
I20260925 15:11:40.017169 22631 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:40.017803 22630 sys_catalog.cc:458] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:40.019436 22635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:40.024636 22635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:40.032938 22635 catalog_manager.cc:1347] Generated new cluster ID: 8d681f1517bc4a9eb1ba6bf89b94dc88
I20260925 15:11:40.033051 22635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:40.040167 22635 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:40.041250 22635 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:40.051959 22635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: Generated new TSK 0
I20260925 15:11:40.052666 22635 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:11:40.067845 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--builtin_ntp_servers=127.4.205.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:40.416095 22648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:40.416359 22648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:40.416510 22648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:40.423844 22648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:40.423954 22648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:40.424130 22648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:40.432806 22648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:40.434065 22648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:40.435693 22648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:40.448522 22654 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:40.448626 22653 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:40.448608 22656 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:40.449331 22648 server_base.cc:1034] running on GCE node
I20260925 15:11:40.449939 22648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:40.450829 22648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:40.452057 22648 hybrid_clock.cc:648] HybridClock initialized: now 1790349100452021 us; error 47 us; skew 500 ppm
I20260925 15:11:40.452353 22648 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:40.454418 22648 webserver.cc:460] Webserver started at http://127.4.205.193:42591/ using document root <none> and password file <none>
I20260925 15:11:40.455056 22648 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:40.455184 22648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:40.455485 22648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:40.458189 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "ef5cc8d6ba9d43d8a389b718af934825"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "22f67bc24858789bbe318bf7005bef71"
server_key_iv: "d4376422b495a5b32b844780f77f7128"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.458921 22648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "ef5cc8d6ba9d43d8a389b718af934825"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "22f67bc24858789bbe318bf7005bef71"
server_key_iv: "d4376422b495a5b32b844780f77f7128"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.463639 22648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:40.466917 22662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.468102 22648 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 15:11:40.468273 22648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "ef5cc8d6ba9d43d8a389b718af934825"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "22f67bc24858789bbe318bf7005bef71"
server_key_iv: "d4376422b495a5b32b844780f77f7128"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.468480 22648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:40.502436 22648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:40.503293 22648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:40.503623 22648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:40.504562 22648 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:40.506098 22648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:40.506220 22648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.506353 22648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:40.506438 22648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.539590 22648 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:33211
I20260925 15:11:40.539672 22774 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:33211 every 8 connection(s)
I20260925 15:11:40.540874 22648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:40.544059 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22648
I20260925 15:11:40.544234 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:40.544641 4919 external_mini_cluster.cc:1428] Setting key 08dc51e8627252b1941ba1dd2a71c55b
I20260925 15:11:40.547304 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--builtin_ntp_servers=127.4.205.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:40.554371 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:40.543326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:52771 (local address 127.4.205.254:46663)
0925 15:11:40.543779 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:40.543785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:40.545089 (+ 1304us) server_negotiation.cc:407] Connection header received
0925 15:11:40.547155 (+ 2066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:40.547161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:40.547243 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:40.547365 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:40.548389 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.549527 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.551092 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.551897 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.552056 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:40.553375 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:40.553384 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:40.553392 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:40.553415 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:40.553466 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:40.553484 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:40.553594 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:40.553815 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:11:40.553920 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":205,"threads_started":1}
I20260925 15:11:40.555408 22778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:40.543924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46663 (local address 127.4.205.193:52771)
0925 15:11:40.544908 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:40.544951 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:40.546910 (+ 1959us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:40.547619 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:40.547626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:40.547707 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:40.548226 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:40.548241 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.549749 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.549753 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:40.550934 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:40.550946 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.552142 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.552152 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:40.552381 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:40.553082 (+ 701us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:40.553084 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:40.553120 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:40.553191 (+ 71us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:40.553620 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:40.553628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:40.553633 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:11:40.553774 (+ 141us) client_negotiation.cc:240] Negotiation successful
0925 15:11:40.554154 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":321,"threads_started":1}
I20260925 15:11:40.556809 22775 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46663
I20260925 15:11:40.557067 22775 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:40.557776 22775 heartbeater.cc:511] Master 127.4.205.254:46663 requested a full tablet report, sending...
I20260925 15:11:40.559602 22589 ts_manager.cc:194] Registered new tserver with Master: ef5cc8d6ba9d43d8a389b718af934825 (127.4.205.193:33211)
I20260925 15:11:40.561017 22589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.193:52771
W20260925 15:11:40.857081 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:40.857380 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:40.857534 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:40.864167 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:40.864272 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:40.864430 22779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:40.872840 22779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:40.874028 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:40.875649 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:40.887495 22784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:40.887552 22787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:40.887629 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:40.888361 22779 server_base.cc:1034] running on GCE node
I20260925 15:11:40.888878 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:40.889748 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:40.890959 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790349100890918 us; error 53 us; skew 500 ppm
I20260925 15:11:40.891243 22779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:40.893450 22779 webserver.cc:460] Webserver started at http://127.4.205.194:43471/ using document root <none> and password file <none>
I20260925 15:11:40.894085 22779 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:40.894219 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:40.894516 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:40.897032 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "6f8ff07556b44ea0951abcb1828ccab3"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "88b148d7dabd2670522121e344f1b2c1"
server_key_iv: "0487fcf977560fe8d349d52d58db22bb"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.897825 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "6f8ff07556b44ea0951abcb1828ccab3"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "88b148d7dabd2670522121e344f1b2c1"
server_key_iv: "0487fcf977560fe8d349d52d58db22bb"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.902334 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:11:40.905534 22793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.906755 22779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 15:11:40.906914 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "6f8ff07556b44ea0951abcb1828ccab3"
format_stamp: "Formatted at 2026-09-25 15:11:40 on dist-test-slave-64fx"
server_key: "88b148d7dabd2670522121e344f1b2c1"
server_key_iv: "0487fcf977560fe8d349d52d58db22bb"
server_key_version: "encryptionkey@0"
I20260925 15:11:40.907105 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:40.921869 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:40.922565 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:40.922839 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:40.923722 22779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:40.925009 22779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:40.925091 22779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.925241 22779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:40.925326 22779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:40.957816 22779 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:42843
I20260925 15:11:40.957901 22905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:42843 every 8 connection(s)
I20260925 15:11:40.959079 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:40.960481 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22779
I20260925 15:11:40.960634 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:40.961231 4919 external_mini_cluster.cc:1428] Setting key a29b62fdf0970c5a780b0bc96edb98eb
I20260925 15:11:40.963881 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--builtin_ntp_servers=127.4.205.212:46253
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:40.971979 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:40.961710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:48521 (local address 127.4.205.254:46663)
0925 15:11:40.961880 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:40.961884 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:40.963480 (+ 1596us) server_negotiation.cc:407] Connection header received
0925 15:11:40.965182 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:40.965190 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:40.965275 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:40.965387 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:40.966456 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.967550 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.968987 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.969694 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.969834 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:40.971157 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:40.971165 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:40.971170 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:40.971204 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:40.971233 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:40.971244 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:40.971311 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:40.971600 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 15:11:40.971683 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:11:40.973826 22908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:40.962344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46663 (local address 127.4.205.194:48521)
0925 15:11:40.963330 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:40.963370 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:40.964926 (+ 1556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:40.965600 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:40.965613 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:40.965712 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:40.966279 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:40.966301 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.967683 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.967687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:40.968849 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:40.968859 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:40.969876 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:40.969880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:40.970058 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:40.970811 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:40.970813 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:40.970849 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:40.970916 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:40.971323 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:40.971336 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:40.971345 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:11:40.971535 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 15:11:40.971922 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":320,"threads_started":1}
I20260925 15:11:40.975116 22906 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46663
I20260925 15:11:40.975376 22906 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:40.976266 22906 heartbeater.cc:511] Master 127.4.205.254:46663 requested a full tablet report, sending...
I20260925 15:11:40.977752 22589 ts_manager.cc:194] Registered new tserver with Master: 6f8ff07556b44ea0951abcb1828ccab3 (127.4.205.194:42843)
I20260925 15:11:40.978811 22589 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.194:48521
W20260925 15:11:41.275799 22909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:41.276084 22909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:41.276247 22909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:41.283527 22909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:41.283655 22909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:41.283828 22909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:41.292930 22909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46253
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:46663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:41.294222 22909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:41.295933 22909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:41.308529 22915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:41.308495 22914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:41.308631 22917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:41.309262 22909 server_base.cc:1034] running on GCE node
I20260925 15:11:41.309893 22909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:41.310783 22909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:41.311995 22909 hybrid_clock.cc:648] HybridClock initialized: now 1790349101311934 us; error 59 us; skew 500 ppm
I20260925 15:11:41.312287 22909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:41.314601 22909 webserver.cc:460] Webserver started at http://127.4.205.195:46733/ using document root <none> and password file <none>
I20260925 15:11:41.315244 22909 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:41.315368 22909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:41.315722 22909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:41.318521 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "55c22788616844bd9e61adb0b79b39b2"
format_stamp: "Formatted at 2026-09-25 15:11:41 on dist-test-slave-64fx"
server_key: "470747c16287aad2256b1aa05e3460fb"
server_key_iv: "bf95f2e879bc9da3d0a797f5e11e5893"
server_key_version: "encryptionkey@0"
I20260925 15:11:41.319243 22909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "55c22788616844bd9e61adb0b79b39b2"
format_stamp: "Formatted at 2026-09-25 15:11:41 on dist-test-slave-64fx"
server_key: "470747c16287aad2256b1aa05e3460fb"
server_key_iv: "bf95f2e879bc9da3d0a797f5e11e5893"
server_key_version: "encryptionkey@0"
I20260925 15:11:41.324110 22909 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 15:11:41.327359 22923 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:41.328598 22909 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:11:41.328766 22909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "55c22788616844bd9e61adb0b79b39b2"
format_stamp: "Formatted at 2026-09-25 15:11:41 on dist-test-slave-64fx"
server_key: "470747c16287aad2256b1aa05e3460fb"
server_key_iv: "bf95f2e879bc9da3d0a797f5e11e5893"
server_key_version: "encryptionkey@0"
I20260925 15:11:41.328982 22909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:41.350623 22909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:41.351400 22909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:41.351707 22909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:41.352599 22909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:41.353991 22909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:41.354102 22909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:41.354233 22909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:41.354316 22909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:41.385879 22909 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:46711
I20260925 15:11:41.385954 23035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:46711 every 8 connection(s)
I20260925 15:11:41.387109 22909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:41.389153 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 22909
I20260925 15:11:41.389331 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:41.389940 4919 external_mini_cluster.cc:1428] Setting key 6d2d6deb48ad80f80f41308a741e4ad1
I20260925 15:11:41.399277 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.389778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:54821 (local address 127.4.205.254:46663)
0925 15:11:41.389988 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.389995 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.391487 (+ 1492us) server_negotiation.cc:407] Connection header received
0925 15:11:41.393190 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.393195 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.393278 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.393393 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:41.394446 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.395160 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.396610 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.397055 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.397184 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.398497 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:41.398503 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:41.398514 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:41.398529 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:41.398558 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:41.398568 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:41.398657 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.398860 (+ 203us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.398971 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:11:41.400061 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.390311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:46663 (local address 127.4.205.195:54821)
0925 15:11:41.391305 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.391352 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.392933 (+ 1581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.393649 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.393657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.393735 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:41.394299 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.394315 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.395305 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.395311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.396476 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.396486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.397266 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.397273 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.397484 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.398196 (+ 712us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:41.398198 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:41.398233 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:41.398301 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:41.398665 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:41.398676 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:41.398684 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.398834 (+ 150us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.399192 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":318,"threads_started":1}
I20260925 15:11:41.401289 23036 heartbeater.cc:347] Connected to a master server at 127.4.205.254:46663
I20260925 15:11:41.401623 23036 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:41.402292 23036 heartbeater.cc:511] Master 127.4.205.254:46663 requested a full tablet report, sending...
I20260925 15:11:41.403735 22588 ts_manager.cc:194] Registered new tserver with Master: 55c22788616844bd9e61adb0b79b39b2 (127.4.205.195:46711)
I20260925 15:11:41.404682 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:41.404935 22588 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.195:54821
I20260925 15:11:41.413933 22777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.408579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.4.205.254:46663)
0925 15:11:41.408760 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.408764 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.409234 (+ 470us) server_negotiation.cc:407] Connection header received
0925 15:11:41.409323 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.409326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.409393 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.409482 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:41.409920 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.410642 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.412000 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.412385 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.412481 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.413384 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:41.413389 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:41.413392 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:41.413407 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:41.413433 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:41.413443 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:41.413507 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.413664 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.413727 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:11:41.417536 22588 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:41.424203 22588 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:41.445647 23052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.436426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56704)
0925 15:11:41.437289 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.437312 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.437398 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.438669 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.438677 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.438716 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:41.439119 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.439130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.441023 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.441029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.442536 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.442548 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.444598 (+ 2050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.444604 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.444864 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.444920 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.445044 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.445230 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"mutex_wait_us":1,"thread_start_us":871,"threads_started":1}
I20260925 15:11:41.445941 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.437088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.4.205.193:33211)
0925 15:11:41.437911 (+ 823us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.437922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.437958 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:11:41.438086 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.438096 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.438396 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.438598 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:41.439357 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.440887 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.442710 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.444085 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.444227 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.444456 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.445433 (+ 977us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.445637 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":502,"threads_started":1}
I20260925 15:11:41.447597 23049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.435198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:42843 (local address 127.0.0.1:33862)
0925 15:11:41.435885 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.435938 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.436131 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.440335 (+ 4204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.440339 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.440360 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:41.440799 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.440814 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.442993 (+ 2179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.442998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.445031 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.445043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.446745 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.446753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.446949 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.447038 (+ 89us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.447114 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.447237 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":91,"thread_start_us":307,"threads_started":1}
I20260925 15:11:41.447789 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.435288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:46711 (local address 127.0.0.1:58656)
0925 15:11:41.436697 (+ 1409us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.436718 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.436841 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.440630 (+ 3789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.440640 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.440665 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:41.441150 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.441162 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.443261 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.443266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.445230 (+ 1964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.445242 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.446888 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.446893 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.447177 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.447207 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.447318 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.447442 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1274,"mutex_wait_us":72,"thread_start_us":1161,"threads_started":1}
I20260925 15:11:41.448064 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.435234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33862 (local address 127.4.205.194:42843)
0925 15:11:41.439771 (+ 4537us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.439783 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.439812 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:11:41.439914 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.439921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.440137 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.440999 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:41.441277 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.442863 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.445235 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.446528 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.446679 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.447454 (+ 775us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.447529 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.447693 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4364,"thread_start_us":4222,"threads_started":1}
I20260925 15:11:41.448603 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.435815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.4.205.195:46711)
0925 15:11:41.440061 (+ 4246us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.440068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.440094 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:11:41.440199 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.440206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.440420 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.441364 (+ 944us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:41.441673 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.443078 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.445451 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.446695 (+ 1244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.446968 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.447229 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.448075 (+ 846us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.448246 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4097,"thread_start_us":4751,"threads_started":1}
I20260925 15:11:41.449658 22710 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 (DEFAULT_TABLE table=test-table [id=0410e8bd7678445e932292e8cfc42d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:41.450798 22710 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d0154c90945ff9ae7500cb8c6b583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:41.451498 22841 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 (DEFAULT_TABLE table=test-table [id=0410e8bd7678445e932292e8cfc42d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:41.451792 22971 tablet_service.cc:1467] Processing CreateTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 (DEFAULT_TABLE table=test-table [id=0410e8bd7678445e932292e8cfc42d6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:41.452760 22841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d0154c90945ff9ae7500cb8c6b583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:41.452966 22971 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e0d0154c90945ff9ae7500cb8c6b583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:41.461366 23054 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Bootstrap starting.
I20260925 15:11:41.464241 23054 tablet_bootstrap.cc:654] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:41.465202 23054 log.cc:824] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:41.466251 23055 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Bootstrap starting.
I20260925 15:11:41.470268 23055 tablet_bootstrap.cc:654] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:41.470279 23056 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Bootstrap starting.
I20260925 15:11:41.471772 23055 log.cc:824] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:41.473446 23056 tablet_bootstrap.cc:654] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:41.474395 23056 log.cc:824] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:41.475100 23054 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: No bootstrap required, opened a new log
I20260925 15:11:41.475325 23055 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: No bootstrap required, opened a new log
I20260925 15:11:41.475404 23054 ts_tablet_manager.cc:1397] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.004s
I20260925 15:11:41.475597 23055 ts_tablet_manager.cc:1397] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 15:11:41.477325 23056 tablet_bootstrap.cc:492] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: No bootstrap required, opened a new log
I20260925 15:11:41.477604 23056 ts_tablet_manager.cc:1397] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 15:11:41.484198 23056 raft_consensus.cc:348] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.484555 23056 raft_consensus.cc:374] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:41.484683 23056 raft_consensus.cc:729] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f8ff07556b44ea0951abcb1828ccab3, State: Initialized, Role: FOLLOWER
I20260925 15:11:41.485258 23056 consensus_queue.cc:260] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.485859 23054 raft_consensus.cc:348] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.485859 23055 raft_consensus.cc:348] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.486251 23055 raft_consensus.cc:374] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:41.486269 23054 raft_consensus.cc:374] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:41.486354 23054 raft_consensus.cc:729] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef5cc8d6ba9d43d8a389b718af934825, State: Initialized, Role: FOLLOWER
I20260925 15:11:41.486368 23055 raft_consensus.cc:729] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c22788616844bd9e61adb0b79b39b2, State: Initialized, Role: FOLLOWER
I20260925 15:11:41.487039 23054 consensus_queue.cc:260] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.487007 23055 consensus_queue.cc:260] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.487645 22906 heartbeater.cc:503] Master 127.4.205.254:46663 was elected leader, sending a full tablet report...
I20260925 15:11:41.488260 23056 ts_tablet_manager.cc:1428] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Time spent starting tablet: real 0.011s user 0.003s sys 0.007s
I20260925 15:11:41.489732 22775 heartbeater.cc:503] Master 127.4.205.254:46663 was elected leader, sending a full tablet report...
I20260925 15:11:41.490556 23054 ts_tablet_manager.cc:1428] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Time spent starting tablet: real 0.015s user 0.010s sys 0.004s
I20260925 15:11:41.491591 23036 heartbeater.cc:503] Master 127.4.205.254:46663 was elected leader, sending a full tablet report...
I20260925 15:11:41.492063 23055 ts_tablet_manager.cc:1428] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260925 15:11:41.543033 22776 tablet.cc:2369] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:11:41.638733 23037 tablet.cc:2369] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:41.643829 23061 raft_consensus.cc:482] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:41.644097 23061 raft_consensus.cc:504] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.645830 23061 leader_election.cc:288] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f8ff07556b44ea0951abcb1828ccab3 (127.4.205.194:42843), ef5cc8d6ba9d43d8a389b718af934825 (127.4.205.193:33211)
I20260925 15:11:41.651853 23038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.646152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:42843 (local address 127.4.205.195:47813)
0925 15:11:41.646327 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.646353 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.646468 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.646978 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.646983 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.647020 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:41.647223 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.647237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.648381 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.648384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.650106 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.650141 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.651363 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.651366 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.651493 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.651515 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.651586 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.651675 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":3}
I20260925 15:11:41.651968 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.646255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:47813 (local address 127.4.205.194:42843)
0925 15:11:41.646457 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.646462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.646482 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:11:41.646603 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.646608 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.646687 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.646798 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:41.647399 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.648270 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.650286 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.651234 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.651387 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.651476 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.651630 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.651744 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:11:41.652864 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.646679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:39025 (local address 127.4.205.193:33211)
0925 15:11:41.646863 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.646871 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.647322 (+ 451us) server_negotiation.cc:407] Connection header received
0925 15:11:41.647466 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.647473 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.647554 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.647646 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:41.648067 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.648779 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.650787 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.651895 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.652034 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.652104 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.652536 (+ 432us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.652649 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:11:41.652866 23063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.646524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.4.205.195:39025)
0925 15:11:41.647199 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.647217 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:41.647335 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:41.647659 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:41.647662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:41.647679 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:41.647933 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.647938 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.648922 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.648928 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.650620 (+ 1692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:41.650630 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.652197 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.652202 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:41.652358 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.652391 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 15:11:41.652501 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:11:41.652604 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":354,"threads_started":1}
I20260925 15:11:41.652797 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0d0154c90945ff9ae7500cb8c6b583" candidate_uuid: "55c22788616844bd9e61adb0b79b39b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8ff07556b44ea0951abcb1828ccab3" is_pre_election: true
I20260925 15:11:41.653302 22861 raft_consensus.cc:2454] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c22788616844bd9e61adb0b79b39b2 in term 0.
I20260925 15:11:41.653851 22730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0d0154c90945ff9ae7500cb8c6b583" candidate_uuid: "55c22788616844bd9e61adb0b79b39b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5cc8d6ba9d43d8a389b718af934825" is_pre_election: true
I20260925 15:11:41.653957 22924 leader_election.cc:302] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c22788616844bd9e61adb0b79b39b2, 6f8ff07556b44ea0951abcb1828ccab3; no voters:
I20260925 15:11:41.654307 23061 raft_consensus.cc:2781] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:41.654305 22730 raft_consensus.cc:2454] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55c22788616844bd9e61adb0b79b39b2 in term 0.
I20260925 15:11:41.654470 23061 raft_consensus.cc:482] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:41.654582 23061 raft_consensus.cc:3037] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:41.656694 23061 raft_consensus.cc:504] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.657265 23061 leader_election.cc:288] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [CANDIDATE]: Term 1 election: Requested vote from peers 6f8ff07556b44ea0951abcb1828ccab3 (127.4.205.194:42843), ef5cc8d6ba9d43d8a389b718af934825 (127.4.205.193:33211)
I20260925 15:11:41.657662 22861 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0d0154c90945ff9ae7500cb8c6b583" candidate_uuid: "55c22788616844bd9e61adb0b79b39b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f8ff07556b44ea0951abcb1828ccab3"
I20260925 15:11:41.657737 22730 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e0d0154c90945ff9ae7500cb8c6b583" candidate_uuid: "55c22788616844bd9e61adb0b79b39b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef5cc8d6ba9d43d8a389b718af934825"
I20260925 15:11:41.657863 22861 raft_consensus.cc:3037] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:41.657927 22730 raft_consensus.cc:3037] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:41.660022 22861 raft_consensus.cc:2454] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c22788616844bd9e61adb0b79b39b2 in term 1.
I20260925 15:11:41.660115 22730 raft_consensus.cc:2454] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55c22788616844bd9e61adb0b79b39b2 in term 1.
I20260925 15:11:41.660403 22924 leader_election.cc:302] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55c22788616844bd9e61adb0b79b39b2, 6f8ff07556b44ea0951abcb1828ccab3; no voters:
I20260925 15:11:41.660646 23061 raft_consensus.cc:2781] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:41.661072 23061 raft_consensus.cc:686] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 1 LEADER]: Becoming Leader. State: Replica: 55c22788616844bd9e61adb0b79b39b2, State: Running, Role: LEADER
I20260925 15:11:41.661598 23061 consensus_queue.cc:237] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } }
I20260925 15:11:41.665951 22587 catalog_manager.cc:5495] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55c22788616844bd9e61adb0b79b39b2 (127.4.205.195). New cstate: current_term: 1 leader_uuid: "55c22788616844bd9e61adb0b79b39b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55c22788616844bd9e61adb0b79b39b2" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 46711 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 } health_report { overall_health: UNKNOWN } } }
I20260925 15:11:41.680502 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.675643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.4.205.195:46711)
0925 15:11:41.675843 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.675849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.675868 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:11:41.675949 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.675952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.676026 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.676167 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:41.676513 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.677199 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.678685 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.679099 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.679187 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.679590 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:41.680082 (+ 492us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:41.680187 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.680266 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.680325 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:11:41.687997 22730 raft_consensus.cc:1261] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 1 FOLLOWER]: Refusing update from remote peer 55c22788616844bd9e61adb0b79b39b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:41.688025 22861 raft_consensus.cc:1261] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 1 FOLLOWER]: Refusing update from remote peer 55c22788616844bd9e61adb0b79b39b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:41.688884 23064 consensus_queue.cc:1035] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f8ff07556b44ea0951abcb1828ccab3" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 42843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:41.689219 23061 consensus_queue.cc:1035] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef5cc8d6ba9d43d8a389b718af934825" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 33211 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:41.703815 23067 mvcc.cc:205] Tried to move back new op lower bound from 7333269920505204736 to 7333269920409509888. Current Snapshot: MvccSnapshot[applied={T|T < 7333269920505204736 or (T in {7333269920505204736})}]
I20260925 15:11:41.717075 22588 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:41.717370 22588 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:41.720675 22588 catalog_manager.cc:5782] T 00000000000000000000000000000000 P da9d0b9d4dab4deb990c44e3fa3fd56a: Sending DeleteTablet for 3 replicas of tablet 5e0d0154c90945ff9ae7500cb8c6b583
I20260925 15:11:41.721796 22971 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:41 UTC) from {username='slave'} at 127.0.0.1:58656
I20260925 15:11:41.721801 22710 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:41 UTC) from {username='slave'} at 127.0.0.1:56704
I20260925 15:11:41.722589 22841 tablet_service.cc:1514] Processing DeleteTablet for tablet 5e0d0154c90945ff9ae7500cb8c6b583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:41 UTC) from {username='slave'} at 127.0.0.1:33862
I20260925 15:11:41.723137 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22648
I20260925 15:11:41.726507 23079 tablet_replica.cc:330] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: stopping tablet replica
I20260925 15:11:41.727015 23079 raft_consensus.cc:2229] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:41.727592 23079 raft_consensus.cc:2258] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:41.729998 23079 ts_tablet_manager.cc:1905] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:41.730602 23078 tablet_replica.cc:330] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: stopping tablet replica
I20260925 15:11:41.730917 23078 raft_consensus.cc:2229] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:41.731401 23078 raft_consensus.cc:2258] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:41.734745 23078 ts_tablet_manager.cc:1905] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:41.739087 23053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.724303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56718 (local address 127.4.205.193:33211)
0925 15:11:41.724459 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.724465 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.725876 (+ 1411us) server_negotiation.cc:407] Connection header received
0925 15:11:41.726072 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.726077 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.726187 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.726294 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:41.732406 (+ 6112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.733531 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.735294 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.735876 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.735991 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.737415 (+ 1424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:41.737426 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:41.737439 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:41.738175 (+ 736us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:41.738232 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:41.738263 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:41.738354 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.738607 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.738750 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:11:41.740850 23077 tablet_replica.cc:330] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825: stopping tablet replica
I20260925 15:11:41.741071 23079 ts_tablet_manager.cc:1918] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:41.741220 23079 log.cc:1196] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/5e0d0154c90945ff9ae7500cb8c6b583
I20260925 15:11:41.741385 22770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:41.741669 23079 ts_tablet_manager.cc:1939] T 5e0d0154c90945ff9ae7500cb8c6b583 P 6f8ff07556b44ea0951abcb1828ccab3: Deleting consensus metadata
I20260925 15:11:41.747790 23078 ts_tablet_manager.cc:1918] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:41.747942 23078 log.cc:1196] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/5e0d0154c90945ff9ae7500cb8c6b583
I20260925 15:11:41.748461 23078 ts_tablet_manager.cc:1939] T 5e0d0154c90945ff9ae7500cb8c6b583 P 55c22788616844bd9e61adb0b79b39b2: Deleting consensus metadata
I20260925 15:11:41.750262 22574 catalog_manager.cc:4841] TS 55c22788616844bd9e61adb0b79b39b2 (127.4.205.195:46711): tablet 5e0d0154c90945ff9ae7500cb8c6b583 (table test-table [id=0410e8bd7678445e932292e8cfc42d6d]) successfully deleted
I20260925 15:11:41.751006 22573 catalog_manager.cc:4841] TS 6f8ff07556b44ea0951abcb1828ccab3 (127.4.205.194:42843): tablet 5e0d0154c90945ff9ae7500cb8c6b583 (table test-table [id=0410e8bd7678445e932292e8cfc42d6d]) successfully deleted
I20260925 15:11:41.828002 23077 raft_consensus.cc:2229] T 5e0d0154c90945ff9ae7500cb8c6b583 P ef5cc8d6ba9d43d8a389b718af934825 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:41.828871 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22648
I20260925 15:11:41.849246 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22779
W20260925 15:11:41.849434 22573 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.205.193:33211 (error 108)
I20260925 15:11:41.852136 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.851592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56732)
0925 15:11:41.851821 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.851945 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260925 15:11:41.852378 22573 catalog_manager.cc:4584] TS ef5cc8d6ba9d43d8a389b718af934825 (127.4.205.193:33211): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5e0d0154c90945ff9ae7500cb8c6b583: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
I20260925 15:11:41.856772 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.850131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33878 (local address 127.4.205.194:42843)
0925 15:11:41.850331 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.850337 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.850488 (+ 151us) server_negotiation.cc:407] Connection header received
0925 15:11:41.850712 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.850718 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.850841 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.850946 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:41.851585 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.852501 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.853932 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.854321 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.854476 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.855542 (+ 1066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:41.855552 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:41.855564 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:41.856090 (+ 526us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:41.856139 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:41.856156 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:41.856270 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.856391 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.856490 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:11:41.857364 22901 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:41.905987 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.905422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56746)
0925 15:11:41.905683 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.905797 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:11:41.907083 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.906701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56762)
0925 15:11:41.906861 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.906941 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 15:11:41.944855 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22779
I20260925 15:11:41.961359 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22909
I20260925 15:11:41.968264 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.967743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56770)
0925 15:11:41.967969 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.968072 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:11:41.968561 23051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.962145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58674 (local address 127.4.205.195:46711)
0925 15:11:41.962324 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:41.962330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:41.962406 (+ 76us) server_negotiation.cc:407] Connection header received
0925 15:11:41.962610 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:41.962624 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:41.962708 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:41.962812 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:41.963217 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.964017 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.965236 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:41.965777 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:41.965909 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:41.967115 (+ 1206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:41.967124 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:41.967136 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:41.967693 (+ 557us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:41.967735 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:41.967754 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:41.967896 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:41.968105 (+ 209us) server_negotiation.cc:299] Negotiation successful
0925 15:11:41.968198 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:11:41.969306 23031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:41.969528 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:41.969117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56780)
0925 15:11:41.969294 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:41.969378 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 15:11:42.050501 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:42.049918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56786)
0925 15:11:42.050143 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:42.050311 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:11:42.051518 23050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:42.051110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:33211 (local address 127.0.0.1:56790)
0925 15:11:42.051303 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:42.051379 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:33211: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 15:11:42.087146 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22909
I20260925 15:11:42.105484 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 22558
I20260925 15:11:42.106488 22619 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:42.214103 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 22558
2026-09-25T15:11:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2718 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:42 dist-test-slave-64fx krb5kdc[23088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:42 dist-test-slave-64fx krb5kdc[23088](info): set up 2 sockets
Sep 25 15:11:42 dist-test-slave-64fx krb5kdc[23088](info): commencing operation
krb5kdc: starting...
W20260925 15:11:43.278082 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:11:43 dist-test-slave-64fx krb5kdc[23088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:11:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:11:43.503975 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44133
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:44133
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:43.820052 23101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:43.820389 23101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:43.820549 23101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:43.827621 23101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:43.827764 23101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:43.827870 23101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:43.827973 23101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:43.828050 23101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:43.837656 23101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:44133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:44133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:43.838891 23101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:43.840180 23101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:43.851442 23106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:43.851645 23107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:43.851641 23109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:43.852407 23101 server_base.cc:1034] running on GCE node
I20260925 15:11:43.853178 23101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:43.854621 23101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:43.855845 23101 hybrid_clock.cc:648] HybridClock initialized: now 1790349103855790 us; error 48 us; skew 500 ppm
I20260925 15:11:43.856185 23101 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:43 dist-test-slave-64fx krb5kdc[23088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:43.859148 23101 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:11:43.860616 23101 webserver.cc:460] Webserver started at http://127.4.205.254:37557/ using document root <none> and password file <none>
I20260925 15:11:43.861295 23101 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:43.861435 23101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:43.861792 23101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:43.864632 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "33c29b30aceb411ca0ac986e92463a7b"
format_stamp: "Formatted at 2026-09-25 15:11:43 on dist-test-slave-64fx"
server_key: "21e611efc165f230e479f5c18b131ff5"
server_key_iv: "a6185f2616b94a051550a56e20981e43"
server_key_version: "encryptionkey@0"
I20260925 15:11:43.865358 23101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "33c29b30aceb411ca0ac986e92463a7b"
format_stamp: "Formatted at 2026-09-25 15:11:43 on dist-test-slave-64fx"
server_key: "21e611efc165f230e479f5c18b131ff5"
server_key_iv: "a6185f2616b94a051550a56e20981e43"
server_key_version: "encryptionkey@0"
I20260925 15:11:43.870271 23101 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:11:43.873379 23116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:43.874651 23101 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:11:43.874817 23101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "33c29b30aceb411ca0ac986e92463a7b"
format_stamp: "Formatted at 2026-09-25 15:11:43 on dist-test-slave-64fx"
server_key: "21e611efc165f230e479f5c18b131ff5"
server_key_iv: "a6185f2616b94a051550a56e20981e43"
server_key_version: "encryptionkey@0"
I20260925 15:11:43.874996 23101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:43.899842 23101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:43.903674 23101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:43.904003 23101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:43.921366 23101 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:44133
I20260925 15:11:43.921399 23167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:44133 every 8 connection(s)
I20260925 15:11:43.922657 23101 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:43.926178 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23101
I20260925 15:11:43.926342 23168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:43.926380 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:43.926937 4919 external_mini_cluster.cc:1428] Setting key 0bcc3bc5eb4fd81ace53dfeba13935df
I20260925 15:11:43.940364 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: Bootstrap starting.
Sep 25 15:11:43 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:43.943943 23168 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:43.944929 23168 log.cc:824] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:43.947860 23168 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: No bootstrap required, opened a new log
I20260925 15:11:43.949416 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:43.928771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34300 (local address 127.4.205.254:44133)
0925 15:11:43.929521 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:43.929531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:43.929616 (+ 85us) server_negotiation.cc:407] Connection header received
0925 15:11:43.931167 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:43.931187 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:43.931545 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:43.931862 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:43.933738 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:43.934900 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:43.936760 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:43.937447 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:43.937658 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:43.942516 (+ 4858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:43.942524 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:43.942536 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:43.942586 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:43.945709 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:43.946486 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:43.946492 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:43.946497 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:43.946644 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:43.947014 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:43.947018 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:43.947020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:43.947444 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:43.947650 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:43.948135 (+ 485us) server_negotiation.cc:299] Negotiation successful
0925 15:11:43.948465 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":352,"threads_started":1}
I20260925 15:11:43.955389 23168 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } }
I20260925 15:11:43.955706 23168 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:43.955822 23168 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c29b30aceb411ca0ac986e92463a7b, State: Initialized, Role: FOLLOWER
I20260925 15:11:43.956327 23168 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } }
I20260925 15:11:43.956539 23168 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:43.956679 23168 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:43.956866 23168 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:43.959040 23168 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } }
I20260925 15:11:43.959503 23168 leader_election.cc:302] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c29b30aceb411ca0ac986e92463a7b; no voters:
I20260925 15:11:43.960047 23168 leader_election.cc:288] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:43.960206 23173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:43.960878 23173 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [term 1 LEADER]: Becoming Leader. State: Replica: 33c29b30aceb411ca0ac986e92463a7b, State: Running, Role: LEADER
I20260925 15:11:43.961443 23173 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } }
I20260925 15:11:43.962138 23168 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:43.964607 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c29b30aceb411ca0ac986e92463a7b. Latest consensus state: current_term: 1 leader_uuid: "33c29b30aceb411ca0ac986e92463a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } } }
I20260925 15:11:43.965034 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:43.964567 23174 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c29b30aceb411ca0ac986e92463a7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c29b30aceb411ca0ac986e92463a7b" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 44133 } } }
I20260925 15:11:43.965531 23174 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:43.966840 23181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:43.972263 23181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:43.980118 23181 catalog_manager.cc:1347] Generated new cluster ID: 1c6d044e968541f484b2b53ee52b5d88
I20260925 15:11:43.980233 23181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:44.010536 23181 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:44.011884 23181 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:44.024659 23181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: Generated new TSK 0
I20260925 15:11:44.025470 23181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:11:44.089768 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:44.421306 23194 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:44.421627 23194 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:44.421804 23194 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:44.428608 23194 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:44.428745 23194 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:44.428900 23194 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:44.438102 23194 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:44.439282 23194 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:44.440548 23194 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:44.454087 23199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:44.454139 23200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:44.454684 23202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:44.455168 23194 server_base.cc:1034] running on GCE node
I20260925 15:11:44.455735 23194 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:44.456543 23194 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:44.457738 23194 hybrid_clock.cc:648] HybridClock initialized: now 1790349104457679 us; error 49 us; skew 500 ppm
I20260925 15:11:44.458043 23194 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:44 dist-test-slave-64fx krb5kdc[23088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:44.461010 23194 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:11:44.462558 23194 webserver.cc:460] Webserver started at http://127.4.205.193:32897/ using document root <none> and password file <none>
I20260925 15:11:44.463232 23194 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:44.463351 23194 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:44.463644 23194 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:44.466372 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "dd5123065ea549d5b873611da0984f67"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "f9d8c5fc5f47316ed15843485373e5d7"
server_key_iv: "7ca37acad9b1bfeaa1ea4dc4d7e14318"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.467142 23194 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "dd5123065ea549d5b873611da0984f67"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "f9d8c5fc5f47316ed15843485373e5d7"
server_key_iv: "7ca37acad9b1bfeaa1ea4dc4d7e14318"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.471767 23194 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260925 15:11:44.474797 23209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:44.476123 23194 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:11:44.476305 23194 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "dd5123065ea549d5b873611da0984f67"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "f9d8c5fc5f47316ed15843485373e5d7"
server_key_iv: "7ca37acad9b1bfeaa1ea4dc4d7e14318"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.476487 23194 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:44.494817 23194 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:44.498389 23194 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:44.498658 23194 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:44.499532 23194 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:44.500759 23194 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:44.500828 23194 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:44.500985 23194 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:44.501076 23194 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:44.533082 23194 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:36785
I20260925 15:11:44.533128 23321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:36785 every 8 connection(s)
I20260925 15:11:44.534426 23194 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:44.535369 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23194
I20260925 15:11:44.535538 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:44.535957 4919 external_mini_cluster.cc:1428] Setting key d3f2efd6756d1b44fb7269627959cffd
Sep 25 15:11:44 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:44.553895 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:44.536907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:44205 (local address 127.4.205.254:44133)
0925 15:11:44.537404 (+ 497us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:44.537410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:44.538569 (+ 1159us) server_negotiation.cc:407] Connection header received
0925 15:11:44.540359 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:44.540363 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:44.540440 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:44.540588 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:44.542147 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:44.542855 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:44.544311 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:44.544725 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:44.544872 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:44.548399 (+ 3527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:44.548403 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:44.548407 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:44.548438 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:44.551014 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:44.551691 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:44.551699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:44.551719 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:44.551826 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:44.552270 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:44.552276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:44.552282 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:44.552525 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:44.552685 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:44.553321 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:11:44.553538 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":251,"threads_started":1}
I20260925 15:11:44.554729 23325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:44.537683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44133 (local address 127.4.205.193:44205)
0925 15:11:44.538431 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:44.538472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:44.540118 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:44.540834 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:44.540842 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:44.541333 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:44.541979 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:44.541996 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:44.542999 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:44.543009 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:44.544164 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:44.544175 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:44.544952 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:44.544960 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:44.545190 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:44.545861 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:44.545896 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:44.548191 (+ 2295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:44.551151 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:44.551160 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:44.551175 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:44.551563 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:44.551938 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:44.551944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:44.551948 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:44.552128 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:44.552653 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:44.552661 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:44.552968 (+ 307us) client_negotiation.cc:769] Sending connection context
0925 15:11:44.553217 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 15:11:44.553709 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":320,"threads_started":1}
I20260925 15:11:44.556248 23322 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44133
I20260925 15:11:44.556529 23322 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:44.557302 23322 heartbeater.cc:511] Master 127.4.205.254:44133 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
I20260925 15:11:44.559573 23133 ts_manager.cc:194] Registered new tserver with Master: dd5123065ea549d5b873611da0984f67 (127.4.205.193:36785)
I20260925 15:11:44.560964 23133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:44205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:11:44.588452 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:44.892529 23328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:44.892829 23328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:44.893009 23328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:44.899909 23328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:44.900060 23328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:44.900241 23328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:44.909312 23328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:44.910440 23328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:44.911685 23328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:44.924535 23333 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:44.924579 23334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:44.925040 23336 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:44.925037 23328 server_base.cc:1034] running on GCE node
I20260925 15:11:44.925813 23328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:44.926795 23328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:44.928059 23328 hybrid_clock.cc:648] HybridClock initialized: now 1790349104928011 us; error 40 us; skew 500 ppm
I20260925 15:11:44.928366 23328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:44 dist-test-slave-64fx krb5kdc[23088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:44.938094 23328 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:11:44.939288 23328 webserver.cc:460] Webserver started at http://127.4.205.194:34813/ using document root <none> and password file <none>
I20260925 15:11:44.939824 23328 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:44.939921 23328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:44.940174 23328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:44.942802 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "34e211f2ab6b49dcb263eec884044f07"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "90740c588f284715d5cfc9ee1012a03f"
server_key_iv: "0342a92a8ac02e6cc6b1e52f105de15b"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.943439 23328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "34e211f2ab6b49dcb263eec884044f07"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "90740c588f284715d5cfc9ee1012a03f"
server_key_iv: "0342a92a8ac02e6cc6b1e52f105de15b"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.948006 23328 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:44.951108 23343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:44.952303 23328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:11:44.952486 23328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "34e211f2ab6b49dcb263eec884044f07"
format_stamp: "Formatted at 2026-09-25 15:11:44 on dist-test-slave-64fx"
server_key: "90740c588f284715d5cfc9ee1012a03f"
server_key_iv: "0342a92a8ac02e6cc6b1e52f105de15b"
server_key_version: "encryptionkey@0"
I20260925 15:11:44.952662 23328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:44.963299 23328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:44.966882 23328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:44.967228 23328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:44.968256 23328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:44.969404 23328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:44.969502 23328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:44.969643 23328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:44.969712 23328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:45.004143 23328 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:45869
I20260925 15:11:45.004177 23455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:45869 every 8 connection(s)
I20260925 15:11:45.005424 23328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:45.007180 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23328
I20260925 15:11:45.007373 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:45.007819 4919 external_mini_cluster.cc:1428] Setting key ba5e2672a5026d3fffe5e3c43a388a15
Sep 25 15:11:45 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349104, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:11:45.035779 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.008903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:57245 (local address 127.4.205.254:44133)
0925 15:11:45.009104 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.009110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.018401 (+ 9291us) server_negotiation.cc:407] Connection header received
0925 15:11:45.020256 (+ 1855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.020262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.020353 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.020468 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:45.022189 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.022908 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.025435 (+ 2527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.025906 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.025999 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.030122 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:45.030127 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:45.030130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:45.030161 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:45.032818 (+ 2657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.033520 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.033531 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.033533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.033706 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.034160 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.034166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.034170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.034437 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:45.034598 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.035234 (+ 636us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.035447 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:11:45.036878 23458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.009478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44133 (local address 127.4.205.194:57245)
0925 15:11:45.018224 (+ 8746us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.018266 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.019974 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.020721 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.020729 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.021256 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:45.022012 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.022039 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.023063 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.023071 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.025283 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.025301 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.026115 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.026123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.026428 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.027189 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:45.027222 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:45.029903 (+ 2681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:45.032953 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:45.032963 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:45.032979 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:45.033397 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:45.033842 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:45.033847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:45.033851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:45.034041 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:45.034558 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:45.034567 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:45.034889 (+ 322us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.035140 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.035620 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8487,"thread_start_us":8234,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
I20260925 15:11:45.038470 23456 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44133
I20260925 15:11:45.038777 23456 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:45.039434 23456 heartbeater.cc:511] Master 127.4.205.254:44133 requested a full tablet report, sending...
I20260925 15:11:45.040890 23133 ts_manager.cc:194] Registered new tserver with Master: 34e211f2ab6b49dcb263eec884044f07 (127.4.205.194:45869)
I20260925 15:11:45.042165 23133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:57245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:11:45.065117 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:46415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:45.374092 23461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:45.374356 23461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:45.374509 23461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:45.381448 23461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:45.381595 23461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:45.381762 23461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:45.390514 23461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:46415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:44133
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:45.391656 23461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:45.392917 23461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:45.404975 23467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:45.404994 23466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:45.405016 23469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:45.405942 23461 server_base.cc:1034] running on GCE node
I20260925 15:11:45.406471 23461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:45.407316 23461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:45.408501 23461 hybrid_clock.cc:648] HybridClock initialized: now 1790349105408439 us; error 56 us; skew 500 ppm
I20260925 15:11:45.408803 23461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:45 dist-test-slave-64fx krb5kdc[23088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:45.411641 23461 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:11:45.412942 23461 webserver.cc:460] Webserver started at http://127.4.205.195:33299/ using document root <none> and password file <none>
I20260925 15:11:45.413620 23461 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:45.413765 23461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:45.414075 23461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:45.416747 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "06e236f7464843629c7af042269a6f92"
format_stamp: "Formatted at 2026-09-25 15:11:45 on dist-test-slave-64fx"
server_key: "17271b6521d4f907a64c6cc9114862bd"
server_key_iv: "381b4f93b5ad62cbefe5ddc1e8d57270"
server_key_version: "encryptionkey@0"
I20260925 15:11:45.417486 23461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "06e236f7464843629c7af042269a6f92"
format_stamp: "Formatted at 2026-09-25 15:11:45 on dist-test-slave-64fx"
server_key: "17271b6521d4f907a64c6cc9114862bd"
server_key_iv: "381b4f93b5ad62cbefe5ddc1e8d57270"
server_key_version: "encryptionkey@0"
I20260925 15:11:45.422269 23461 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:11:45.425375 23476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:45.426540 23461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:11:45.426704 23461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "06e236f7464843629c7af042269a6f92"
format_stamp: "Formatted at 2026-09-25 15:11:45 on dist-test-slave-64fx"
server_key: "17271b6521d4f907a64c6cc9114862bd"
server_key_iv: "381b4f93b5ad62cbefe5ddc1e8d57270"
server_key_version: "encryptionkey@0"
I20260925 15:11:45.426899 23461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:45.437525 23461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:45.441092 23461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:45.441365 23461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:45.442307 23461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:45.443493 23461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:45.443594 23461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:45.443742 23461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:45.443813 23461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:45.476274 23461 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:35557
I20260925 15:11:45.476323 23588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:35557 every 8 connection(s)
I20260925 15:11:45.477525 23461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:45.485505 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23461
I20260925 15:11:45.485711 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:45.486243 4919 external_mini_cluster.cc:1428] Setting key 3d0d314f0bfed32d8c6646e33b624897
Sep 25 15:11:45 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349105, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:45.497116 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.480088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:60405 (local address 127.4.205.254:44133)
0925 15:11:45.480254 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.480258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.481622 (+ 1364us) server_negotiation.cc:407] Connection header received
0925 15:11:45.483428 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.483432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.483496 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.483585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:45.484954 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.485922 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.488100 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.488733 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.488863 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.492149 (+ 3286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:45.492154 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:45.492157 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:45.492186 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:45.494702 (+ 2516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.495336 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.495340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.495342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.495461 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.495802 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.495805 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.495807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.496051 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:45.496161 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.496740 (+ 579us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.496886 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:11:45.498018 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.480630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:44133 (local address 127.4.205.195:60405)
0925 15:11:45.481435 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.481489 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.483206 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.483782 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.483789 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.484252 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:45.484781 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.484797 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.486064 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.486071 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.487952 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.487966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.488946 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.488952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.489221 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.489911 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:45.489932 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:45.491978 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:45.494808 (+ 2830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:45.494814 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:45.494824 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:45.495200 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:45.495551 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:45.495554 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:45.495557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:45.495716 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:45.496154 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:45.496159 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:45.496450 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.496661 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.496985 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":364,"threads_started":1}
I20260925 15:11:45.499207 23589 heartbeater.cc:347] Connected to a master server at 127.4.205.254:44133
I20260925 15:11:45.499517 23589 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:45.500288 23589 heartbeater.cc:511] Master 127.4.205.254:44133 requested a full tablet report, sending...
I20260925 15:11:45.501734 23133 ts_manager.cc:194] Registered new tserver with Master: 06e236f7464843629c7af042269a6f92 (127.4.205.195:35557)
I20260925 15:11:45.502492 23133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:60405
I20260925 15:11:45.512835 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:45.528244 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.516578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34302 (local address 127.4.205.254:44133)
0925 15:11:45.516785 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.516793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.517154 (+ 361us) server_negotiation.cc:407] Connection header received
0925 15:11:45.517277 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.517281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.517336 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.517413 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:45.518719 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.519386 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.520637 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.521000 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.521103 (+ 103us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.523542 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:45.523547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:45.523550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:45.523577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:45.526002 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.526646 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.526650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.526652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.526764 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.527093 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.527096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.527097 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.527336 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:45.527522 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.527784 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.527899 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:11:45.531931 23133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:45.538612 23133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:45.563792 23322 heartbeater.cc:503] Master 127.4.205.254:44133 was elected leader, sending a full tablet report...
I20260925 15:11:45.570832 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.549760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38396 (local address 127.4.205.195:35557)
0925 15:11:45.560921 (+ 11161us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.560931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.561502 (+ 571us) server_negotiation.cc:407] Connection header received
0925 15:11:45.561950 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.561958 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.562210 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.562449 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:45.563789 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.565368 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.568224 (+ 2856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.569651 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.569784 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.570023 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.570311 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.570472 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10978,"thread_start_us":10821,"threads_started":1}
I20260925 15:11:45.570860 23603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.549401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:35557 (local address 127.0.0.1:38396)
0925 15:11:45.561350 (+ 11949us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.561392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.561690 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.562893 (+ 1203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.562904 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.562954 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:45.563525 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.563544 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.565628 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.565634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.568072 (+ 2438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.568086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.569849 (+ 1763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.569857 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.570062 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.570113 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.570244 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.570427 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11682,"mutex_wait_us":24,"spinlock_wait_cycles":2176,"thread_start_us":11700,"threads_started":1}
I20260925 15:11:45.571885 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.549823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:45869 (local address 127.0.0.1:60898)
0925 15:11:45.561350 (+ 11527us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.561392 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.561690 (+ 298us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.564006 (+ 2316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.564012 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.564034 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:45.564456 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.564471 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.566926 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.566932 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.569086 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.569107 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.571088 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.571096 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.571281 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.571316 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.571408 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.571532 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11402,"mutex_wait_us":48,"spinlock_wait_cycles":6784,"thread_start_us":11083,"threads_started":1}
I20260925 15:11:45.572110 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.549927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.4.205.194:45869)
0925 15:11:45.560547 (+ 10620us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.560561 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.563086 (+ 2525us) server_negotiation.cc:407] Connection header received
0925 15:11:45.563197 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.563220 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.563479 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.563658 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:45.564644 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.566793 (+ 2149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.569257 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.570630 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.570757 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.570947 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.571468 (+ 521us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.571652 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10379,"thread_start_us":10236,"threads_started":1}
I20260925 15:11:45.574795 23524 tablet_service.cc:1467] Processing CreateTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba (DEFAULT_TABLE table=test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:45.575116 23391 tablet_service.cc:1467] Processing CreateTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba (DEFAULT_TABLE table=test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:45.576036 23524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edec10dca3243ec9d3e704e9bf2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:45.576249 23391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edec10dca3243ec9d3e704e9bf2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:45.582115 23605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.562020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:36785 (local address 127.0.0.1:60022)
0925 15:11:45.563169 (+ 1149us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.563186 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.563297 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.565634 (+ 2337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.565638 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.565664 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:45.566062 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.566079 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.567700 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.567708 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.569932 (+ 2224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.569945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.571988 (+ 2043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.571993 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.581652 (+ 9659us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.581677 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.581759 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.581855 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1013,"mutex_wait_us":4,"thread_start_us":868,"threads_started":1}
I20260925 15:11:45.582305 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.562069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60022 (local address 127.4.205.193:36785)
0925 15:11:45.564455 (+ 2386us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.564462 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.564491 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:11:45.565159 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.565168 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.565374 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.565639 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:45.566268 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.567564 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.570120 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.571522 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.571642 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.571853 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.581829 (+ 9976us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.581996 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2206,"thread_start_us":2044,"threads_started":1}
I20260925 15:11:45.585841 23257 tablet_service.cc:1467] Processing CreateTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba (DEFAULT_TABLE table=test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:45.587188 23257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5edec10dca3243ec9d3e704e9bf2c3ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:45.587652 23607 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Bootstrap starting.
I20260925 15:11:45.588876 23608 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Bootstrap starting.
I20260925 15:11:45.591187 23607 tablet_bootstrap.cc:654] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:45.592291 23608 tablet_bootstrap.cc:654] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:45.592471 23607 log.cc:824] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:45.593721 23608 log.cc:824] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:45.595615 23607 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: No bootstrap required, opened a new log
I20260925 15:11:45.595866 23607 ts_tablet_manager.cc:1397] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260925 15:11:45.597988 23608 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: No bootstrap required, opened a new log
I20260925 15:11:45.598233 23608 ts_tablet_manager.cc:1397] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 15:11:45.599105 23611 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Bootstrap starting.
I20260925 15:11:45.602288 23611 tablet_bootstrap.cc:654] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:45.603196 23611 log.cc:824] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:45.605469 23611 tablet_bootstrap.cc:492] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: No bootstrap required, opened a new log
I20260925 15:11:45.605751 23611 ts_tablet_manager.cc:1397] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260925 15:11:45.606901 23607 raft_consensus.cc:348] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.607337 23607 raft_consensus.cc:374] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:45.607443 23607 raft_consensus.cc:729] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34e211f2ab6b49dcb263eec884044f07, State: Initialized, Role: FOLLOWER
I20260925 15:11:45.608032 23607 consensus_queue.cc:260] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.608525 23608 raft_consensus.cc:348] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.609004 23608 raft_consensus.cc:374] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:45.609112 23608 raft_consensus.cc:729] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e236f7464843629c7af042269a6f92, State: Initialized, Role: FOLLOWER
I20260925 15:11:45.609189 23456 heartbeater.cc:503] Master 127.4.205.254:44133 was elected leader, sending a full tablet report...
I20260925 15:11:45.609825 23607 ts_tablet_manager.cc:1428] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Time spent starting tablet: real 0.014s user 0.007s sys 0.006s
I20260925 15:11:45.609704 23608 consensus_queue.cc:260] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.610810 23589 heartbeater.cc:503] Master 127.4.205.254:44133 was elected leader, sending a full tablet report...
I20260925 15:11:45.611411 23608 ts_tablet_manager.cc:1428] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260925 15:11:45.616264 23611 raft_consensus.cc:348] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.616580 23611 raft_consensus.cc:374] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:45.616698 23611 raft_consensus.cc:729] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd5123065ea549d5b873611da0984f67, State: Initialized, Role: FOLLOWER
I20260925 15:11:45.617188 23611 consensus_queue.cc:260] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.618541 23611 ts_tablet_manager.cc:1428] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Time spent starting tablet: real 0.013s user 0.007s sys 0.004s
W20260925 15:11:45.729270 23590 tablet.cc:2369] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:45.753711 23613 raft_consensus.cc:482] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:45.754005 23613 raft_consensus.cc:504] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.755800 23613 leader_election.cc:288] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e236f7464843629c7af042269a6f92 (127.4.205.195:35557), dd5123065ea549d5b873611da0984f67 (127.4.205.193:36785)
W20260925 15:11:45.757282 23457 tablet.cc:2369] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:45.762478 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.756337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:54471 (local address 127.4.205.193:36785)
0925 15:11:45.756574 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.756583 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.757128 (+ 545us) server_negotiation.cc:407] Connection header received
0925 15:11:45.757347 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.757360 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.757459 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.757548 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:45.758295 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.759016 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.760740 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.761531 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.761738 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.761840 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.762102 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.762240 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 15:11:45.762552 23616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.756074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:36785 (local address 127.4.205.194:54471)
0925 15:11:45.756982 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.757011 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.757192 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.757603 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.757609 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.757657 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:45.758122 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.758134 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.759163 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.759170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.760587 (+ 1417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.760598 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.761760 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.761767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.761928 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.761951 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.762075 (+ 124us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.762216 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":786,"thread_start_us":534,"threads_started":1}
I20260925 15:11:45.763299 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.756127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:50853 (local address 127.4.205.195:35557)
0925 15:11:45.756377 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.756387 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.757934 (+ 1547us) server_negotiation.cc:407] Connection header received
0925 15:11:45.758141 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.758147 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.758248 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.758325 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:45.758822 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.759582 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.761249 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.762277 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.762441 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.762515 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.762910 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.763017 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:11:45.763595 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edec10dca3243ec9d3e704e9bf2c3ba" candidate_uuid: "34e211f2ab6b49dcb263eec884044f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5123065ea549d5b873611da0984f67" is_pre_election: true
I20260925 15:11:45.763933 23617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.756088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:35557 (local address 127.4.205.194:50853)
0925 15:11:45.757805 (+ 1717us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:45.757840 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:45.758009 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:45.758336 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:45.758340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:45.758360 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:45.758669 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.758676 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.759733 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.759740 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.761084 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:45.761095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.762480 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.762486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:45.762669 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.762703 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:11:45.763472 (+ 769us) client_negotiation.cc:240] Negotiation successful
0925 15:11:45.763600 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1567,"thread_start_us":772,"threads_started":1}
I20260925 15:11:45.764196 23277 raft_consensus.cc:2454] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e211f2ab6b49dcb263eec884044f07 in term 0.
I20260925 15:11:45.764868 23345 leader_election.cc:302] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 34e211f2ab6b49dcb263eec884044f07, dd5123065ea549d5b873611da0984f67; no voters:
I20260925 15:11:45.764839 23544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edec10dca3243ec9d3e704e9bf2c3ba" candidate_uuid: "34e211f2ab6b49dcb263eec884044f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e236f7464843629c7af042269a6f92" is_pre_election: true
I20260925 15:11:45.765229 23613 raft_consensus.cc:2781] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:45.765252 23544 raft_consensus.cc:2454] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 34e211f2ab6b49dcb263eec884044f07 in term 0.
I20260925 15:11:45.765334 23613 raft_consensus.cc:482] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:45.765455 23613 raft_consensus.cc:3037] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:45.767719 23613 raft_consensus.cc:504] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.768347 23613 leader_election.cc:288] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [CANDIDATE]: Term 1 election: Requested vote from peers 06e236f7464843629c7af042269a6f92 (127.4.205.195:35557), dd5123065ea549d5b873611da0984f67 (127.4.205.193:36785)
I20260925 15:11:45.768651 23544 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edec10dca3243ec9d3e704e9bf2c3ba" candidate_uuid: "34e211f2ab6b49dcb263eec884044f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e236f7464843629c7af042269a6f92"
I20260925 15:11:45.768854 23544 raft_consensus.cc:3037] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:45.769003 23277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5edec10dca3243ec9d3e704e9bf2c3ba" candidate_uuid: "34e211f2ab6b49dcb263eec884044f07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd5123065ea549d5b873611da0984f67"
I20260925 15:11:45.769184 23277 raft_consensus.cc:3037] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:45.771066 23544 raft_consensus.cc:2454] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e211f2ab6b49dcb263eec884044f07 in term 1.
I20260925 15:11:45.771369 23277 raft_consensus.cc:2454] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 34e211f2ab6b49dcb263eec884044f07 in term 1.
I20260925 15:11:45.771435 23344 leader_election.cc:302] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e236f7464843629c7af042269a6f92, 34e211f2ab6b49dcb263eec884044f07; no voters:
I20260925 15:11:45.771744 23613 raft_consensus.cc:2781] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:45.772181 23613 raft_consensus.cc:686] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 1 LEADER]: Becoming Leader. State: Replica: 34e211f2ab6b49dcb263eec884044f07, State: Running, Role: LEADER
I20260925 15:11:45.772619 23613 consensus_queue.cc:237] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } }
I20260925 15:11:45.776576 23131 catalog_manager.cc:5495] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 reported cstate change: term changed from 0 to 1, leader changed from <none> to 34e211f2ab6b49dcb263eec884044f07 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "34e211f2ab6b49dcb263eec884044f07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "34e211f2ab6b49dcb263eec884044f07" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 45869 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 } health_report { overall_health: UNKNOWN } } }
W20260925 15:11:45.786583 23323 tablet.cc:2369] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:45.809433 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.804053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.4.205.194:45869)
0925 15:11:45.804254 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.804260 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.804280 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:11:45.804448 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.804452 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.804525 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.804685 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:45.805080 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.805863 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.807317 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.807690 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.807831 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.808407 (+ 576us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:45.808873 (+ 466us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:45.809058 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.809163 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.809207 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:11:45.816907 23277 raft_consensus.cc:1261] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 1 FOLLOWER]: Refusing update from remote peer 34e211f2ab6b49dcb263eec884044f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:45.816991 23544 raft_consensus.cc:1261] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 1 FOLLOWER]: Refusing update from remote peer 34e211f2ab6b49dcb263eec884044f07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:45.817875 23618 consensus_queue.cc:1035] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd5123065ea549d5b873611da0984f67" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 36785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:45.818234 23613 consensus_queue.cc:1035] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e236f7464843629c7af042269a6f92" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 35557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:45.828565 23621 mvcc.cc:205] Tried to move back new op lower bound from 7333269937417068544 to 7333269937247952896. Current Snapshot: MvccSnapshot[applied={T|T < 7333269937417068544}]
I20260925 15:11:45.829873 23623 mvcc.cc:205] Tried to move back new op lower bound from 7333269937417068544 to 7333269937247952896. Current Snapshot: MvccSnapshot[applied={T|T < 7333269937417068544}]
I20260925 15:11:45.847451 23133 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:45.847734 23133 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:45.850997 23133 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 33c29b30aceb411ca0ac986e92463a7b: Sending DeleteTablet for 3 replicas of tablet 5edec10dca3243ec9d3e704e9bf2c3ba
I20260925 15:11:45.852061 23524 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:45 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:38396
I20260925 15:11:45.852254 23391 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:45 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60898
I20260925 15:11:45.852382 23257 tablet_service.cc:1514] Processing DeleteTablet for tablet 5edec10dca3243ec9d3e704e9bf2c3ba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:45 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:60022
I20260925 15:11:45.856755 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23194
I20260925 15:11:45.866173 23632 tablet_replica.cc:330] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: stopping tablet replica
I20260925 15:11:45.866590 23632 raft_consensus.cc:2229] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:45.867275 23632 raft_consensus.cc:2258] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:45.868068 23633 tablet_replica.cc:330] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: stopping tablet replica
I20260925 15:11:45.868441 23633 raft_consensus.cc:2229] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:45.868929 23633 raft_consensus.cc:2258] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:45.869627 23631 tablet_replica.cc:330] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: stopping tablet replica
I20260925 15:11:45.869971 23631 raft_consensus.cc:2229] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:45.870388 23631 raft_consensus.cc:2258] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:45.871728 23632 ts_tablet_manager.cc:1905] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:45.872169 23633 ts_tablet_manager.cc:1905] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:45.879072 23632 ts_tablet_manager.cc:1918] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:45.879262 23632 log.cc:1196] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/5edec10dca3243ec9d3e704e9bf2c3ba
I20260925 15:11:45.879365 23631 ts_tablet_manager.cc:1905] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:45.879982 23632 ts_tablet_manager.cc:1939] T 5edec10dca3243ec9d3e704e9bf2c3ba P 34e211f2ab6b49dcb263eec884044f07: Deleting consensus metadata
Sep 25 15:11:45 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:11:45.881814 23118 catalog_manager.cc:4841] TS 34e211f2ab6b49dcb263eec884044f07 (127.4.205.194:45869): tablet 5edec10dca3243ec9d3e704e9bf2c3ba (table test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]) successfully deleted
I20260925 15:11:45.883736 23633 ts_tablet_manager.cc:1918] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:45.883880 23633 log.cc:1196] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/5edec10dca3243ec9d3e704e9bf2c3ba
I20260925 15:11:45.884445 23633 ts_tablet_manager.cc:1939] T 5edec10dca3243ec9d3e704e9bf2c3ba P dd5123065ea549d5b873611da0984f67: Deleting consensus metadata
I20260925 15:11:45.886200 23631 ts_tablet_manager.cc:1918] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:45.886262 23118 catalog_manager.cc:4841] TS dd5123065ea549d5b873611da0984f67 (127.4.205.193:36785): tablet 5edec10dca3243ec9d3e704e9bf2c3ba (table test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]) successfully deleted
I20260925 15:11:45.886340 23631 log.cc:1196] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/5edec10dca3243ec9d3e704e9bf2c3ba
I20260925 15:11:45.886826 23631 ts_tablet_manager.cc:1939] T 5edec10dca3243ec9d3e704e9bf2c3ba P 06e236f7464843629c7af042269a6f92: Deleting consensus metadata
I20260925 15:11:45.888427 23117 catalog_manager.cc:4841] TS 06e236f7464843629c7af042269a6f92 (127.4.205.195:35557): tablet 5edec10dca3243ec9d3e704e9bf2c3ba (table test-table [id=e253e3e8d8f94de19d3dc328a3adb6be]) successfully deleted
I20260925 15:11:45.889314 23606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:45.858414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60028 (local address 127.4.205.193:36785)
0925 15:11:45.858581 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:45.858587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:45.870082 (+ 11495us) server_negotiation.cc:407] Connection header received
0925 15:11:45.870424 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:45.870429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:45.870517 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:45.870599 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:45.872515 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.873646 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.875352 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:45.875992 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:45.876088 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:45.882401 (+ 6313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:45.882413 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:45.882429 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:45.882467 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:45.886568 (+ 4101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.887346 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.887355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.887359 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.887489 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:45.887866 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:45.887872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:45.887877 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:45.888324 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:45.888679 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:45.888895 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:11:45.889038 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:11:45.889966 23317 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:46.024312 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23194
I20260925 15:11:46.047194 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23328
Sep 25 15:11:46 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:11:46.064216 23604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:46.047962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60908 (local address 127.4.205.194:45869)
0925 15:11:46.048136 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:46.048142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:46.048212 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:11:46.048412 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:46.048426 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:46.048487 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:46.048605 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:46.050063 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.050881 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.052187 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.052640 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.052784 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:46.057247 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:46.057265 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:46.057284 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:46.057324 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:46.061129 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:46.062032 (+ 903us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:46.062042 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:46.062046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:46.062224 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:46.062638 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:46.062643 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:46.062647 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:46.063054 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:46.063367 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:46.063593 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:11:46.063813 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:11:46.064997 23451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:46.178252 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23328
I20260925 15:11:46.196993 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23461
Sep 25 15:11:46 dist-test-slave-64fx krb5kdc[23088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:11:46.211189 23602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:46.197765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.4.205.195:35557)
0925 15:11:46.197935 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:46.197940 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:46.198113 (+ 173us) server_negotiation.cc:407] Connection header received
0925 15:11:46.198249 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:46.198253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:46.198319 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:46.198396 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:46.199615 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.200282 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.201461 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.201899 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.201991 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:46.206090 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:46.206105 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:46.206119 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:46.206152 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:46.208751 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:46.209338 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:46.209344 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:46.209346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:46.209472 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:46.209897 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:46.209900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:46.209902 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:46.210234 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:46.210437 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:46.210701 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:11:46.210848 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:11:46.211822 23584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:46.298254 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23461
I20260925 15:11:46.314270 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23101
I20260925 15:11:46.315258 23163 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:46.423488 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23101
2026-09-25T15:11:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4206 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:11:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:46Z Disabled control of system clock
I20260925 15:11:46.493772 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35521
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:35521
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:46.802217 23642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:46.802493 23642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:46.802567 23642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:46.809583 23642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:46.809679 23642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:46.809715 23642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:46.809769 23642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:46.809811 23642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:46.818311 23642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:35521
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:46.819344 23642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:46.821071 23642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:46.831772 23648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:46.832017 23647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:46.831813 23650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:46.832791 23642 server_base.cc:1034] running on GCE node
I20260925 15:11:46.833540 23642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:46.834810 23642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:46.835994 23642 hybrid_clock.cc:648] HybridClock initialized: now 1790349106835932 us; error 57 us; skew 500 ppm
I20260925 15:11:46.836308 23642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:46.838512 23642 webserver.cc:460] Webserver started at http://127.4.205.254:39941/ using document root <none> and password file <none>
I20260925 15:11:46.839131 23642 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:46.839236 23642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:46.839572 23642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:46.842353 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "008ad1e6f1aa4bbcabb1f24100184565"
format_stamp: "Formatted at 2026-09-25 15:11:46 on dist-test-slave-64fx"
server_key: "d96a23c6e685981f175c4da6a939a69a"
server_key_iv: "6b41014af34a36beba2cb81e928d8fef"
server_key_version: "encryptionkey@0"
I20260925 15:11:46.843115 23642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "008ad1e6f1aa4bbcabb1f24100184565"
format_stamp: "Formatted at 2026-09-25 15:11:46 on dist-test-slave-64fx"
server_key: "d96a23c6e685981f175c4da6a939a69a"
server_key_iv: "6b41014af34a36beba2cb81e928d8fef"
server_key_version: "encryptionkey@0"
I20260925 15:11:46.847596 23642 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:46.850551 23656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:46.851783 23642 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:11:46.851965 23642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "008ad1e6f1aa4bbcabb1f24100184565"
format_stamp: "Formatted at 2026-09-25 15:11:46 on dist-test-slave-64fx"
server_key: "d96a23c6e685981f175c4da6a939a69a"
server_key_iv: "6b41014af34a36beba2cb81e928d8fef"
server_key_version: "encryptionkey@0"
I20260925 15:11:46.852156 23642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:46.877556 23642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:46.878335 23642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:46.878600 23642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:46.896066 23642 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:35521
I20260925 15:11:46.896085 23707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:35521 every 8 connection(s)
I20260925 15:11:46.897362 23642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:46.900879 23708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:46.901400 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23642
I20260925 15:11:46.901624 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:46.902084 4919 external_mini_cluster.cc:1428] Setting key f34009ecccafb2353d76678c83138cb0
I20260925 15:11:46.912744 23708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: Bootstrap starting.
I20260925 15:11:46.915310 23711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:46.903942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45452 (local address 127.4.205.254:35521)
0925 15:11:46.904878 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:46.904887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:46.904928 (+ 41us) server_negotiation.cc:407] Connection header received
0925 15:11:46.906882 (+ 1954us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:46.906903 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:46.907335 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:46.907750 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:46.908320 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.909626 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.911031 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:46.911701 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:46.911887 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:46.913103 (+ 1216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:46.913111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:46.913124 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:46.913698 (+ 574us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:46.913778 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:46.913807 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:46.913933 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:46.914133 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:11:46.914424 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":331,"threads_started":1}
I20260925 15:11:46.916189 23708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:46.917248 23708 log.cc:824] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:46.919888 23708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: No bootstrap required, opened a new log
I20260925 15:11:46.926739 23708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } }
I20260925 15:11:46.927075 23708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:46.927184 23708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 008ad1e6f1aa4bbcabb1f24100184565, State: Initialized, Role: FOLLOWER
I20260925 15:11:46.927719 23708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } }
I20260925 15:11:46.927945 23708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:46.928071 23708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:46.928251 23708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:46.930315 23708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } }
I20260925 15:11:46.930795 23708 leader_election.cc:302] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 008ad1e6f1aa4bbcabb1f24100184565; no voters:
I20260925 15:11:46.931335 23708 leader_election.cc:288] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:46.931497 23713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:46.932230 23713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [term 1 LEADER]: Becoming Leader. State: Replica: 008ad1e6f1aa4bbcabb1f24100184565, State: Running, Role: LEADER
I20260925 15:11:46.932799 23713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } }
I20260925 15:11:46.933459 23708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:46.935436 23714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "008ad1e6f1aa4bbcabb1f24100184565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } } }
I20260925 15:11:46.935580 23715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 008ad1e6f1aa4bbcabb1f24100184565. Latest consensus state: current_term: 1 leader_uuid: "008ad1e6f1aa4bbcabb1f24100184565" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "008ad1e6f1aa4bbcabb1f24100184565" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35521 } } }
I20260925 15:11:46.935914 23714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:46.935907 23715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:46.938150 23720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:46.943482 23720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:46.952085 23720 catalog_manager.cc:1347] Generated new cluster ID: f058f287e6b74eee9b1b80a2a3ade344
I20260925 15:11:46.952178 23720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:46.962693 23720 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:46.963940 23720 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:46.968984 23720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: Generated new TSK 0
I20260925 15:11:46.969857 23720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:11:46.975531 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--builtin_ntp_servers=127.4.205.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:47.322211 23732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:47.322480 23732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:47.322621 23732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:47.329353 23732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:47.329471 23732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:47.329668 23732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:47.338398 23732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:47.339541 23732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:47.341127 23732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:47.353415 23737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:47.353478 23740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:47.353453 23738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:47.354591 23732 server_base.cc:1034] running on GCE node
I20260925 15:11:47.355089 23732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:47.355967 23732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:47.357172 23732 hybrid_clock.cc:648] HybridClock initialized: now 1790349107357121 us; error 43 us; skew 500 ppm
I20260925 15:11:47.357470 23732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:47.359521 23732 webserver.cc:460] Webserver started at http://127.4.205.193:42517/ using document root <none> and password file <none>
I20260925 15:11:47.360184 23732 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:47.360308 23732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:47.360602 23732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:47.363279 23732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "410cf85d28c74a6eaa16ccf3471d2bb5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "cc47fe55699874a75471434c99f0a9a7"
server_key_iv: "fce68de86ffab123559b21fcd9965590"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.364012 23732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "410cf85d28c74a6eaa16ccf3471d2bb5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "cc47fe55699874a75471434c99f0a9a7"
server_key_iv: "fce68de86ffab123559b21fcd9965590"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.368564 23732 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:11:47.371734 23746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.372869 23732 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:11:47.373040 23732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "410cf85d28c74a6eaa16ccf3471d2bb5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "cc47fe55699874a75471434c99f0a9a7"
server_key_iv: "fce68de86ffab123559b21fcd9965590"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.373257 23732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:47.387970 23732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:47.388655 23732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:47.388978 23732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:47.389917 23732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:47.391214 23732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:47.391331 23732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.391461 23732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:47.391547 23732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.421924 23732 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:40657
I20260925 15:11:47.421995 23858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:40657 every 8 connection(s)
I20260925 15:11:47.423043 23732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:47.431404 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23732
I20260925 15:11:47.431565 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:47.431996 4919 external_mini_cluster.cc:1428] Setting key e66dd47f43b25e8d7e5b6966b3da838d
I20260925 15:11:47.434566 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--builtin_ntp_servers=127.4.205.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:47.434784 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:47.425271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:59553 (local address 127.4.205.254:35521)
0925 15:11:47.425713 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:47.425717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:47.426739 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 15:11:47.428586 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:47.428592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:47.428692 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:47.428800 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:47.429860 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.430584 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.432253 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.432618 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.432748 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:47.433982 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:47.433988 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:47.433993 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:47.434012 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:47.434064 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:47.434076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:47.434179 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:47.434387 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 15:11:47.434469 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":175,"threads_started":1}
I20260925 15:11:47.435598 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:47.425846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35521 (local address 127.4.205.193:59553)
0925 15:11:47.426604 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:47.426642 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:47.428327 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:47.429005 (+ 678us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:47.429012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:47.429088 (+ 76us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:47.429705 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:47.429722 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.430732 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.430739 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:47.432119 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:47.432130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.432818 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.432824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:47.433019 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:47.433723 (+ 704us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:47.433726 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:47.433758 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:47.433818 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:47.434191 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:47.434204 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:47.434216 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 15:11:47.434360 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 15:11:47.434733 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":228,"threads_started":1}
I20260925 15:11:47.437165 23859 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35521
I20260925 15:11:47.437470 23859 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:47.438237 23859 heartbeater.cc:511] Master 127.4.205.254:35521 requested a full tablet report, sending...
I20260925 15:11:47.440236 23673 ts_manager.cc:194] Registered new tserver with Master: 410cf85d28c74a6eaa16ccf3471d2bb5 (127.4.205.193:40657)
I20260925 15:11:47.442369 23673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.193:59553
W20260925 15:11:47.751443 23863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:47.751688 23863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:47.751840 23863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:47.758885 23863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:47.758989 23863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:47.759163 23863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:47.767613 23863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:47.768714 23863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:47.770406 23863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:47.782367 23871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:47.782440 23869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:47.782462 23868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:47.783242 23863 server_base.cc:1034] running on GCE node
I20260925 15:11:47.783771 23863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:47.784552 23863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:47.785775 23863 hybrid_clock.cc:648] HybridClock initialized: now 1790349107785717 us; error 42 us; skew 500 ppm
I20260925 15:11:47.786064 23863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:47.788036 23863 webserver.cc:460] Webserver started at http://127.4.205.194:41001/ using document root <none> and password file <none>
I20260925 15:11:47.788666 23863 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:47.788798 23863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:47.789094 23863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:47.791772 23863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "c83a1fab52734d5fde13493a24a4df20"
server_key_iv: "7e035961dd23280f2ee511f980915bb5"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.792501 23863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "c83a1fab52734d5fde13493a24a4df20"
server_key_iv: "7e035961dd23280f2ee511f980915bb5"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.797448 23863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260925 15:11:47.800439 23877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.801644 23863 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:11:47.801805 23863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5"
format_stamp: "Formatted at 2026-09-25 15:11:47 on dist-test-slave-64fx"
server_key: "c83a1fab52734d5fde13493a24a4df20"
server_key_iv: "7e035961dd23280f2ee511f980915bb5"
server_key_version: "encryptionkey@0"
I20260925 15:11:47.801997 23863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:47.818099 23863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:47.818775 23863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:47.819047 23863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:47.819916 23863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:47.821202 23863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:47.821314 23863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.821434 23863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:47.821517 23863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:47.853494 23863 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:46783
I20260925 15:11:47.853648 23989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:46783 every 8 connection(s)
I20260925 15:11:47.854823 23863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:47.860461 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23863
I20260925 15:11:47.860641 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:47.861083 4919 external_mini_cluster.cc:1428] Setting key e210358178596775f43963100e8ef50a
I20260925 15:11:47.863686 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--builtin_ntp_servers=127.4.205.212:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:47.866551 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:47.857170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:44825 (local address 127.4.205.254:35521)
0925 15:11:47.857356 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:47.857359 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:47.858590 (+ 1231us) server_negotiation.cc:407] Connection header received
0925 15:11:47.860166 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:47.860172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:47.860264 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:47.860379 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:47.861472 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.862379 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.864000 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.864391 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.864519 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:47.865832 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:47.865838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:47.865844 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:47.865867 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:47.865894 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:47.865905 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:47.865987 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:47.866198 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 15:11:47.866282 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:11:47.867372 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:47.857654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35521 (local address 127.4.205.194:44825)
0925 15:11:47.858459 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:47.858495 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:47.859903 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:47.860538 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:47.860546 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:47.860669 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:47.861308 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:47.861325 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.862516 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.862519 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:47.863809 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:47.863827 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:47.864589 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:47.864598 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:47.864822 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:47.865498 (+ 676us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:47.865501 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:47.865533 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:47.865654 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:47.865999 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:47.866010 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:47.866018 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 15:11:47.866185 (+ 167us) client_negotiation.cc:240] Negotiation successful
0925 15:11:47.866497 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":239,"threads_started":1}
I20260925 15:11:47.868655 23990 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35521
I20260925 15:11:47.868943 23990 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:47.869776 23990 heartbeater.cc:511] Master 127.4.205.254:35521 requested a full tablet report, sending...
I20260925 15:11:47.871101 23673 ts_manager.cc:194] Registered new tserver with Master: c489ea5b6a724a8fb7f779ceddc4dcd5 (127.4.205.194:46783)
I20260925 15:11:47.871877 23673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.194:44825
W20260925 15:11:48.180487 23993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:48.180730 23993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:48.180896 23993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:48.187789 23993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:48.187888 23993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:48.188024 23993 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:48.196756 23993 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:35521
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:48.198019 23993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:48.199735 23993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:48.212410 23998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:48.212360 24001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:48.212445 23999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:48.213482 23993 server_base.cc:1034] running on GCE node
I20260925 15:11:48.214084 23993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:48.214962 23993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:48.216152 23993 hybrid_clock.cc:648] HybridClock initialized: now 1790349108216093 us; error 58 us; skew 500 ppm
I20260925 15:11:48.216471 23993 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:48.218780 23993 webserver.cc:460] Webserver started at http://127.4.205.195:35533/ using document root <none> and password file <none>
I20260925 15:11:48.219453 23993 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:48.219564 23993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:48.219882 23993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:48.222743 23993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "0207aed9337f40698039c1b37c9dc63a"
format_stamp: "Formatted at 2026-09-25 15:11:48 on dist-test-slave-64fx"
server_key: "7f9b773b02decebabfc577873e80d784"
server_key_iv: "7b8063a78f8793a8728b5bab8dc4c098"
server_key_version: "encryptionkey@0"
I20260925 15:11:48.223520 23993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "0207aed9337f40698039c1b37c9dc63a"
format_stamp: "Formatted at 2026-09-25 15:11:48 on dist-test-slave-64fx"
server_key: "7f9b773b02decebabfc577873e80d784"
server_key_iv: "7b8063a78f8793a8728b5bab8dc4c098"
server_key_version: "encryptionkey@0"
I20260925 15:11:48.228292 23993 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:11:48.231532 24007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:48.232805 23993 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:11:48.232965 23993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "0207aed9337f40698039c1b37c9dc63a"
format_stamp: "Formatted at 2026-09-25 15:11:48 on dist-test-slave-64fx"
server_key: "7f9b773b02decebabfc577873e80d784"
server_key_iv: "7b8063a78f8793a8728b5bab8dc4c098"
server_key_version: "encryptionkey@0"
I20260925 15:11:48.233172 23993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:48.246654 23993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:48.247378 23993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:48.247663 23993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:48.248577 23993 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:48.249950 23993 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:48.250067 23993 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:48.250203 23993 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:48.250288 23993 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:48.282603 23993 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:40917
I20260925 15:11:48.282652 24119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:40917 every 8 connection(s)
I20260925 15:11:48.283911 23993 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:48.287477 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 23993
I20260925 15:11:48.287643 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:48.288156 4919 external_mini_cluster.cc:1428] Setting key 55b15d1128f4e49095ef5dad14aafdae
I20260925 15:11:48.295914 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.286362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:41331 (local address 127.4.205.254:35521)
0925 15:11:48.286528 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.286532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.287900 (+ 1368us) server_negotiation.cc:407] Connection header received
0925 15:11:48.290072 (+ 2172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.290076 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.290137 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.290220 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:48.291398 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.292153 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.293694 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.294090 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.294177 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.295308 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:48.295313 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:48.295316 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:48.295332 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:48.295364 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:48.295376 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:48.295442 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.295618 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.295691 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:11:48.296869 24122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.286779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:35521 (local address 127.4.205.195:41331)
0925 15:11:48.287748 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.287794 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.289791 (+ 1997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.290462 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.290473 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.290581 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:48.291257 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.291274 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.292310 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.292314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.293500 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.293511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.294248 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.294252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.294439 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.295055 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:48.295058 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:48.295092 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:48.295158 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:48.295453 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:48.295460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:48.295465 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.295602 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.295857 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":326,"threads_started":1}
I20260925 15:11:48.298151 24120 heartbeater.cc:347] Connected to a master server at 127.4.205.254:35521
I20260925 15:11:48.298422 24120 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:48.299070 24120 heartbeater.cc:511] Master 127.4.205.254:35521 requested a full tablet report, sending...
I20260925 15:11:48.300312 23673 ts_manager.cc:194] Registered new tserver with Master: 0207aed9337f40698039c1b37c9dc63a (127.4.205.195:40917)
I20260925 15:11:48.301083 23673 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.195:41331
I20260925 15:11:48.303231 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:48.312187 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.307160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45458 (local address 127.4.205.254:35521)
0925 15:11:48.307336 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.307340 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.307775 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:11:48.307881 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.307884 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.307943 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.308014 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:48.308386 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.309000 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.310286 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.310637 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.310725 (+ 88us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.311642 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:48.311646 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:48.311650 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:48.311665 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:48.311694 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:48.311705 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:48.311810 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.311952 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.311992 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:11:48.315538 23673 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45458:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:48.321933 23673 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:48.352195 24132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.333287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43358)
0925 15:11:48.343466 (+ 10179us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.343489 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.343645 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.344829 (+ 1184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.344837 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.344880 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:48.345230 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.345244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.347165 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.347170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.349309 (+ 2139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.349325 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.351246 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.351252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.351446 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.351494 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.351628 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.351799 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10004,"thread_start_us":8921,"threads_started":1}
I20260925 15:11:48.352211 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.333519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.4.205.193:40657)
0925 15:11:48.342024 (+ 8505us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.342037 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.343805 (+ 1768us) server_negotiation.cc:407] Connection header received
0925 15:11:48.343958 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.343970 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.344289 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.344514 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:48.345406 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.347001 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.349551 (+ 2550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.351043 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.351188 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.351432 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.351690 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.351855 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8335,"thread_start_us":8183,"threads_started":1}
I20260925 15:11:48.353060 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.332615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:40917 (local address 127.0.0.1:39982)
0925 15:11:48.342700 (+ 10085us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.342760 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.342966 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.345844 (+ 2878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.345849 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.345877 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:48.346280 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.346292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.348272 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.348277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.350488 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.350503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.352062 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.352069 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.352574 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.352623 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.352728 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.352822 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9756,"mutex_wait_us":97,"thread_start_us":10484,"threads_started":1}
I20260925 15:11:48.353238 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.332906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39982 (local address 127.4.205.195:40917)
0925 15:11:48.345177 (+ 12271us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.345186 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.345218 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:11:48.345310 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.345320 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.345580 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.346479 (+ 899us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:48.346720 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.348144 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.350682 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.351847 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.352264 (+ 417us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.352546 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.352767 (+ 221us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.352920 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12098,"thread_start_us":12712,"threads_started":1}
I20260925 15:11:48.356108 24137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.343464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46783 (local address 127.0.0.1:43088)
0925 15:11:48.346262 (+ 2798us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.346275 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.346408 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.346865 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.346869 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.346985 (+ 116us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:48.347339 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.347348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.348600 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.348605 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.353357 (+ 4752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.353371 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.355440 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.355446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.355633 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.355660 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.355737 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.355837 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2685,"mutex_wait_us":3,"thread_start_us":2540,"threads_started":1}
I20260925 15:11:48.356397 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.343737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43088 (local address 127.4.205.194:46783)
0925 15:11:48.345496 (+ 1759us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.345506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.346371 (+ 865us) server_negotiation.cc:407] Connection header received
0925 15:11:48.346555 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.346560 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.346705 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.346877 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:48.347505 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.348463 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.353652 (+ 5189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.354967 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.355087 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.355315 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.355810 (+ 495us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.356015 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612,"thread_start_us":1502,"threads_started":1}
I20260925 15:11:48.356524 24055 tablet_service.cc:1467] Processing CreateTablet for tablet 5197da487f69402aa7e4d90559b32153 (DEFAULT_TABLE table=test-table [id=5572ea43e5a04382b7a80a5155ce8131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:48.356387 23794 tablet_service.cc:1467] Processing CreateTablet for tablet 5197da487f69402aa7e4d90559b32153 (DEFAULT_TABLE table=test-table [id=5572ea43e5a04382b7a80a5155ce8131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:48.358013 24055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5197da487f69402aa7e4d90559b32153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:48.358062 23794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5197da487f69402aa7e4d90559b32153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:48.359668 23925 tablet_service.cc:1467] Processing CreateTablet for tablet 5197da487f69402aa7e4d90559b32153 (DEFAULT_TABLE table=test-table [id=5572ea43e5a04382b7a80a5155ce8131]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:48.360842 23925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5197da487f69402aa7e4d90559b32153. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:48.368743 24138 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Bootstrap starting.
I20260925 15:11:48.371759 24138 tablet_bootstrap.cc:654] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:48.372977 24138 log.cc:824] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:48.374109 24139 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Bootstrap starting.
I20260925 15:11:48.375685 24141 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Bootstrap starting.
I20260925 15:11:48.377854 24139 tablet_bootstrap.cc:654] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:48.378885 24141 tablet_bootstrap.cc:654] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:48.379006 24139 log.cc:824] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:48.380002 24141 log.cc:824] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:48.381388 24138 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: No bootstrap required, opened a new log
I20260925 15:11:48.381686 24138 ts_tablet_manager.cc:1397] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20260925 15:11:48.381717 24139 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: No bootstrap required, opened a new log
I20260925 15:11:48.381964 24139 ts_tablet_manager.cc:1397] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:11:48.382824 24141 tablet_bootstrap.cc:492] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: No bootstrap required, opened a new log
I20260925 15:11:48.383105 24141 ts_tablet_manager.cc:1397] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260925 15:11:48.389221 24139 raft_consensus.cc:348] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.389592 24139 raft_consensus.cc:374] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:48.389706 24139 raft_consensus.cc:729] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 410cf85d28c74a6eaa16ccf3471d2bb5, State: Initialized, Role: FOLLOWER
I20260925 15:11:48.390281 24139 consensus_queue.cc:260] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.391301 23859 heartbeater.cc:503] Master 127.4.205.254:35521 was elected leader, sending a full tablet report...
I20260925 15:11:48.392079 24138 raft_consensus.cc:348] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.392585 24138 raft_consensus.cc:374] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:48.392709 24138 raft_consensus.cc:729] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0207aed9337f40698039c1b37c9dc63a, State: Initialized, Role: FOLLOWER
I20260925 15:11:48.393112 24139 ts_tablet_manager.cc:1428] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260925 15:11:48.393392 24138 consensus_queue.cc:260] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.393311 24141 raft_consensus.cc:348] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.393738 24141 raft_consensus.cc:374] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:48.393841 24141 raft_consensus.cc:729] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c489ea5b6a724a8fb7f779ceddc4dcd5, State: Initialized, Role: FOLLOWER
I20260925 15:11:48.394691 24141 consensus_queue.cc:260] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.399246 23990 heartbeater.cc:503] Master 127.4.205.254:35521 was elected leader, sending a full tablet report...
I20260925 15:11:48.399741 24120 heartbeater.cc:503] Master 127.4.205.254:35521 was elected leader, sending a full tablet report...
I20260925 15:11:48.399870 24141 ts_tablet_manager.cc:1428] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260925 15:11:48.401091 24138 ts_tablet_manager.cc:1428] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Time spent starting tablet: real 0.019s user 0.006s sys 0.011s
W20260925 15:11:48.425007 23860 tablet.cc:2369] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:11:48.535614 24121 tablet.cc:2369] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 15:11:48.606884 23991 tablet.cc:2369] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:48.670223 24144 raft_consensus.cc:482] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:48.670508 24144 raft_consensus.cc:504] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.672585 24144 leader_election.cc:288] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0207aed9337f40698039c1b37c9dc63a (127.4.205.195:40917), c489ea5b6a724a8fb7f779ceddc4dcd5 (127.4.205.194:46783)
I20260925 15:11:48.680815 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.674097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:36509 (local address 127.4.205.194:46783)
0925 15:11:48.674279 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.674284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.674913 (+ 629us) server_negotiation.cc:407] Connection header received
0925 15:11:48.675100 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.675108 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.675201 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.675312 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:48.675918 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.676877 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.678875 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.679767 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.679939 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.680007 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.680385 (+ 378us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.680526 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:11:48.680989 24148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.673832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:46783 (local address 127.4.205.193:36509)
0925 15:11:48.674779 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.674801 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.674941 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.675337 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.675341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.675364 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:48.675737 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.675749 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.677000 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.677008 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.678688 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.678703 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.679953 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.679961 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.680189 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.680242 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.680345 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.680511 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":808,"mutex_wait_us":2,"thread_start_us":590,"threads_started":1}
I20260925 15:11:48.681759 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.673047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:45687 (local address 127.4.205.195:40917)
0925 15:11:48.673278 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.673285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.673807 (+ 522us) server_negotiation.cc:407] Connection header received
0925 15:11:48.674123 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.674131 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.674214 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.674277 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:48.675147 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.676313 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.679501 (+ 3188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.680579 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.680730 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.680794 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.681355 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.681484 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:11:48.682415 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.672995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:40917 (local address 127.4.205.193:45687)
0925 15:11:48.673675 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.673691 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:48.673941 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:48.674502 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:48.674525 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:48.674557 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:48.674954 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.674965 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.676456 (+ 1491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.676464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.679296 (+ 2832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:48.679315 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.680956 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.680964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:48.681156 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.681194 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 15:11:48.681957 (+ 763us) client_negotiation.cc:240] Negotiation successful
0925 15:11:48.682099 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"mutex_wait_us":37,"thread_start_us":306,"threads_started":1}
I20260925 15:11:48.682324 23945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5197da487f69402aa7e4d90559b32153" candidate_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" is_pre_election: true
I20260925 15:11:48.682977 23945 raft_consensus.cc:2454] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410cf85d28c74a6eaa16ccf3471d2bb5 in term 0.
I20260925 15:11:48.683642 23748 leader_election.cc:302] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 410cf85d28c74a6eaa16ccf3471d2bb5, c489ea5b6a724a8fb7f779ceddc4dcd5; no voters:
I20260925 15:11:48.683527 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5197da487f69402aa7e4d90559b32153" candidate_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0207aed9337f40698039c1b37c9dc63a" is_pre_election: true
I20260925 15:11:48.684054 24144 raft_consensus.cc:2781] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:48.684085 24075 raft_consensus.cc:2454] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 410cf85d28c74a6eaa16ccf3471d2bb5 in term 0.
I20260925 15:11:48.684214 24144 raft_consensus.cc:482] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:48.684314 24144 raft_consensus.cc:3037] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:48.686722 24144 raft_consensus.cc:504] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.687381 24144 leader_election.cc:288] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [CANDIDATE]: Term 1 election: Requested vote from peers 0207aed9337f40698039c1b37c9dc63a (127.4.205.195:40917), c489ea5b6a724a8fb7f779ceddc4dcd5 (127.4.205.194:46783)
I20260925 15:11:48.687839 24075 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5197da487f69402aa7e4d90559b32153" candidate_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0207aed9337f40698039c1b37c9dc63a"
I20260925 15:11:48.687943 23945 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5197da487f69402aa7e4d90559b32153" candidate_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5"
I20260925 15:11:48.688067 24075 raft_consensus.cc:3037] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:48.688138 23945 raft_consensus.cc:3037] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:48.691287 24075 raft_consensus.cc:2454] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410cf85d28c74a6eaa16ccf3471d2bb5 in term 1.
I20260925 15:11:48.691296 23945 raft_consensus.cc:2454] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 410cf85d28c74a6eaa16ccf3471d2bb5 in term 1.
I20260925 15:11:48.691709 23748 leader_election.cc:302] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0207aed9337f40698039c1b37c9dc63a, 410cf85d28c74a6eaa16ccf3471d2bb5; no voters:
I20260925 15:11:48.692000 24144 raft_consensus.cc:2781] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:48.692530 24144 raft_consensus.cc:686] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 1 LEADER]: Becoming Leader. State: Replica: 410cf85d28c74a6eaa16ccf3471d2bb5, State: Running, Role: LEADER
I20260925 15:11:48.693356 24144 consensus_queue.cc:237] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } }
I20260925 15:11:48.698958 23672 catalog_manager.cc:5495] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 410cf85d28c74a6eaa16ccf3471d2bb5 (127.4.205.193). New cstate: current_term: 1 leader_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "410cf85d28c74a6eaa16ccf3471d2bb5" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 40657 } health_report { overall_health: HEALTHY } } }
I20260925 15:11:48.798285 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.793336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43374 (local address 127.4.205.193:40657)
0925 15:11:48.793592 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.793599 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.793627 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:11:48.793798 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.793802 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.793893 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.794010 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:48.794321 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.794947 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.796204 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.796622 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.796718 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.797156 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:48.797681 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:48.797806 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.797913 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.797968 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 15:11:48.806160 24075 raft_consensus.cc:1261] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 1 FOLLOWER]: Refusing update from remote peer 410cf85d28c74a6eaa16ccf3471d2bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:48.806160 23945 raft_consensus.cc:1261] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 1 FOLLOWER]: Refusing update from remote peer 410cf85d28c74a6eaa16ccf3471d2bb5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:48.807075 24149 consensus_queue.cc:1035] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0207aed9337f40698039c1b37c9dc63a" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 40917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:48.807500 24144 consensus_queue.cc:1035] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c489ea5b6a724a8fb7f779ceddc4dcd5" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 46783 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:48.818684 24153 mvcc.cc:205] Tried to move back new op lower bound from 7333269949659639808 to 7333269949212098560. Current Snapshot: MvccSnapshot[applied={T|T < 7333269949659639808}]
I20260925 15:11:48.830856 23673 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:48.831111 23673 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45458:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:48.834671 23673 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 008ad1e6f1aa4bbcabb1f24100184565: Sending DeleteTablet for 3 replicas of tablet 5197da487f69402aa7e4d90559b32153
I20260925 15:11:48.835614 24055 tablet_service.cc:1514] Processing DeleteTablet for tablet 5197da487f69402aa7e4d90559b32153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:48 UTC) from {username='slave'} at 127.0.0.1:39982
I20260925 15:11:48.835739 23925 tablet_service.cc:1514] Processing DeleteTablet for tablet 5197da487f69402aa7e4d90559b32153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:48 UTC) from {username='slave'} at 127.0.0.1:43088
I20260925 15:11:48.836371 23794 tablet_service.cc:1514] Processing DeleteTablet for tablet 5197da487f69402aa7e4d90559b32153 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:48 UTC) from {username='slave'} at 127.0.0.1:43358
I20260925 15:11:48.837163 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23732
I20260925 15:11:48.855248 24162 tablet_replica.cc:330] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: stopping tablet replica
I20260925 15:11:48.855455 24163 tablet_replica.cc:330] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5: stopping tablet replica
I20260925 15:11:48.855882 24163 raft_consensus.cc:2229] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:48.855883 24162 raft_consensus.cc:2229] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:48.856380 24162 raft_consensus.cc:2258] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:48.857311 24163 raft_consensus.cc:2258] T 5197da487f69402aa7e4d90559b32153 P 410cf85d28c74a6eaa16ccf3471d2bb5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:48.859462 24162 ts_tablet_manager.cc:1905] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:48.861198 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.839732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.4.205.193:40657)
0925 15:11:48.839930 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.839938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.849552 (+ 9614us) server_negotiation.cc:407] Connection header received
0925 15:11:48.850022 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.850031 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.850141 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.850235 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:48.851029 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.852417 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.857208 (+ 4791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.857888 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.857998 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.859486 (+ 1488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:48.859496 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:48.859511 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:48.860249 (+ 738us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:48.860306 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:48.860345 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:48.860418 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.860717 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.860817 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:11:48.862413 23854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:48.866837 24161 tablet_replica.cc:330] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: stopping tablet replica
I20260925 15:11:48.867412 24161 raft_consensus.cc:2229] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:48.868057 24161 raft_consensus.cc:2258] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:48.868432 24162 ts_tablet_manager.cc:1918] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:48.868615 24162 log.cc:1196] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/5197da487f69402aa7e4d90559b32153
I20260925 15:11:48.869158 24162 ts_tablet_manager.cc:1939] T 5197da487f69402aa7e4d90559b32153 P c489ea5b6a724a8fb7f779ceddc4dcd5: Deleting consensus metadata
I20260925 15:11:48.871069 23658 catalog_manager.cc:4841] TS c489ea5b6a724a8fb7f779ceddc4dcd5 (127.4.205.194:46783): tablet 5197da487f69402aa7e4d90559b32153 (table test-table [id=5572ea43e5a04382b7a80a5155ce8131]) successfully deleted
I20260925 15:11:48.871402 24161 ts_tablet_manager.cc:1905] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:48.878068 24161 ts_tablet_manager.cc:1918] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:48.878274 24161 log.cc:1196] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/5197da487f69402aa7e4d90559b32153
I20260925 15:11:48.878741 24161 ts_tablet_manager.cc:1939] T 5197da487f69402aa7e4d90559b32153 P 0207aed9337f40698039c1b37c9dc63a: Deleting consensus metadata
I20260925 15:11:48.880481 23658 catalog_manager.cc:4841] TS 0207aed9337f40698039c1b37c9dc63a (127.4.205.195:40917): tablet 5197da487f69402aa7e4d90559b32153 (table test-table [id=5572ea43e5a04382b7a80a5155ce8131]) successfully deleted
I20260925 15:11:48.969866 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23732
W20260925 15:11:48.987144 23660 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.4.205.193:40657 (error 108)
I20260925 15:11:48.987391 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23863
I20260925 15:11:48.989174 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.988387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43390)
0925 15:11:48.988808 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:48.988952 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":188,"threads_started":1}
W20260925 15:11:48.989451 23660 catalog_manager.cc:4584] TS 410cf85d28c74a6eaa16ccf3471d2bb5 (127.4.205.193:40657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5197da487f69402aa7e4d90559b32153: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
I20260925 15:11:48.994539 24136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:48.988188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.4.205.194:46783)
0925 15:11:48.988404 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:48.988408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:48.988458 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:11:48.988732 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:48.988737 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:48.988845 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:48.988962 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:48.989332 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.990212 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.991471 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:48.991869 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:48.992001 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:48.993009 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:48.993018 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:48.993031 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:48.993782 (+ 751us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:48.993829 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:48.993848 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:48.993967 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:48.994109 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:11:48.994211 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:11:48.995095 23985 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:49.042884 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.042442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43396)
0925 15:11:49.042651 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.042738 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:11:49.043939 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.043595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43402)
0925 15:11:49.043728 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.043797 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:11:49.083840 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23863
I20260925 15:11:49.100471 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23993
I20260925 15:11:49.104614 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.104119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43418)
0925 15:11:49.104338 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.104437 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 15:11:49.105831 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.105292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43428)
0925 15:11:49.105436 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.105503 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 15:11:49.107544 24133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.101295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39996 (local address 127.4.205.195:40917)
0925 15:11:49.101470 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:49.101475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:49.101493 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:11:49.101651 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:49.101655 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:49.101753 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:49.101839 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:49.102180 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:49.102881 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:49.104363 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:49.104989 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:49.105115 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:49.106350 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:49.106359 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:49.106374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:49.106946 (+ 572us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:49.106993 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:49.107017 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:49.107102 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:49.107215 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:11:49.107300 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:11:49.108156 24115 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:49.186615 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.186156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43436)
0925 15:11:49.186392 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.186475 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 15:11:49.187691 24167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:49.187324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:40657 (local address 127.0.0.1:43440)
0925 15:11:49.187499 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:49.187564 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.193:40657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 15:11:49.196432 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23993
I20260925 15:11:49.211964 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 23642
I20260925 15:11:49.212966 23703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:49.320057 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 23642
2026-09-25T15:11:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2897 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:49 dist-test-slave-64fx krb5kdc[24173](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:49 dist-test-slave-64fx krb5kdc[24173](info): set up 2 sockets
Sep 25 15:11:49 dist-test-slave-64fx krb5kdc[24173](info): commencing operation
krb5kdc: starting...
W20260925 15:11:50.414024 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.008s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:11:50 dist-test-slave-64fx krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:11:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:11:50.628017 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33969
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33969
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:50.935308 24186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:50.935581 24186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:50.935702 24186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:50.942742 24186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:50.942862 24186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:50.942940 24186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:50.943034 24186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:50.943127 24186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:50.952647 24186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33969
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:50.953941 24186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:50.955200 24186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:50.966655 24192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:50.966620 24194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:50.966686 24191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:50.967375 24186 server_base.cc:1034] running on GCE node
I20260925 15:11:50.968125 24186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:50.969468 24186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:50.970682 24186 hybrid_clock.cc:648] HybridClock initialized: now 1790349110970658 us; error 56 us; skew 500 ppm
I20260925 15:11:50.970993 24186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:50 dist-test-slave-64fx krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:50.981874 24186 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:11:50.983412 24186 webserver.cc:460] Webserver started at http://127.4.205.254:44569/ using document root <none> and password file <none>
I20260925 15:11:50.984081 24186 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:50.984226 24186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:50.984527 24186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:50.987399 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d3ed2beecd2e4e15b4d9948b7cedc604"
format_stamp: "Formatted at 2026-09-25 15:11:50 on dist-test-slave-64fx"
server_key: "e2b884bb59e6832f2b9a6c25d1b13767"
server_key_iv: "23165b4296cd533dfbe6e24a59397c9d"
server_key_version: "encryptionkey@0"
I20260925 15:11:50.988157 24186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d3ed2beecd2e4e15b4d9948b7cedc604"
format_stamp: "Formatted at 2026-09-25 15:11:50 on dist-test-slave-64fx"
server_key: "e2b884bb59e6832f2b9a6c25d1b13767"
server_key_iv: "23165b4296cd533dfbe6e24a59397c9d"
server_key_version: "encryptionkey@0"
I20260925 15:11:50.992899 24186 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:11:50.996052 24201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:50.997386 24186 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 15:11:50.997613 24186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d3ed2beecd2e4e15b4d9948b7cedc604"
format_stamp: "Formatted at 2026-09-25 15:11:50 on dist-test-slave-64fx"
server_key: "e2b884bb59e6832f2b9a6c25d1b13767"
server_key_iv: "23165b4296cd533dfbe6e24a59397c9d"
server_key_version: "encryptionkey@0"
I20260925 15:11:50.997800 24186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:51.019016 24186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:51.022732 24186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:51.023020 24186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:51.040831 24186 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33969
I20260925 15:11:51.040859 24252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33969 every 8 connection(s)
I20260925 15:11:51.042146 24186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:51.045578 24253 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:51.049480 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24186
I20260925 15:11:51.049695 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:51.050197 4919 external_mini_cluster.cc:1428] Setting key c892ae9173cca90501b0460ffb9b1d4d
I20260925 15:11:51.058156 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: Bootstrap starting.
I20260925 15:11:51.061771 24253 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:51.063020 24253 log.cc:824] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:11:51 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:51.066102 24253 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: No bootstrap required, opened a new log
I20260925 15:11:51.073513 24256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:51.051924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52560 (local address 127.4.205.254:33969)
0925 15:11:51.052658 (+ 734us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:51.052669 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:51.052705 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:11:51.054198 (+ 1493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:51.054218 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:51.054592 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:51.054889 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:51.056597 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.057873 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.059607 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.060231 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.060356 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:51.065326 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:51.065338 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:51.065355 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:51.065400 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:51.069294 (+ 3894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:51.069959 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:51.069966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:51.069973 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:51.070186 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:51.070553 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:51.070558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:51.070562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:51.071191 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:51.071493 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:51.071808 (+ 315us) server_negotiation.cc:299] Negotiation successful
0925 15:11:51.072254 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":362,"threads_started":1}
I20260925 15:11:51.076016 24253 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } }
I20260925 15:11:51.076313 24253 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:51.076398 24253 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3ed2beecd2e4e15b4d9948b7cedc604, State: Initialized, Role: FOLLOWER
I20260925 15:11:51.076833 24253 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } }
I20260925 15:11:51.077026 24253 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:51.077118 24253 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:51.077235 24253 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:51.079303 24253 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } }
I20260925 15:11:51.079730 24253 leader_election.cc:302] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3ed2beecd2e4e15b4d9948b7cedc604; no voters:
I20260925 15:11:51.080252 24253 leader_election.cc:288] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:51.080384 24258 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:51.081473 24258 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [term 1 LEADER]: Becoming Leader. State: Replica: d3ed2beecd2e4e15b4d9948b7cedc604, State: Running, Role: LEADER
I20260925 15:11:51.082069 24258 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } }
I20260925 15:11:51.082258 24253 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:51.084937 24260 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3ed2beecd2e4e15b4d9948b7cedc604. Latest consensus state: current_term: 1 leader_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } } }
I20260925 15:11:51.085299 24260 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:51.086030 24259 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3ed2beecd2e4e15b4d9948b7cedc604" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33969 } } }
I20260925 15:11:51.086396 24259 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:51.094108 24264 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:51.099177 24264 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:51.107770 24264 catalog_manager.cc:1347] Generated new cluster ID: 166af0c55e354f88935073cd14adebf2
I20260925 15:11:51.107887 24264 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:51.119517 24264 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:51.120759 24264 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:51.134635 24264 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: Generated new TSK 0
I20260925 15:11:51.135396 24264 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.4.205.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
Entry for principal kudu/127.4.205.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab.
I20260925 15:11:51.208856 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:51.523690 24279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:51.523962 24279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:51.524128 24279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:51.530540 24279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:51.530658 24279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:51.530827 24279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:51.539279 24279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.193.keytab
--principal=kudu/127.4.205.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:51.540324 24279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:51.541527 24279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:51.554128 24284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:51.554133 24285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:51.554174 24287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:51.554816 24279 server_base.cc:1034] running on GCE node
I20260925 15:11:51.555320 24279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:51.556079 24279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:51.557253 24279 hybrid_clock.cc:648] HybridClock initialized: now 1790349111557219 us; error 24 us; skew 500 ppm
I20260925 15:11:51.557512 24279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:51 dist-test-slave-64fx krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349111, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:51.560222 24279 init.cc:377] Logged in from keytab as kudu/127.4.205.193@KRBTEST.COM (short username kudu)
I20260925 15:11:51.561709 24279 webserver.cc:460] Webserver started at http://127.4.205.193:36751/ using document root <none> and password file <none>
I20260925 15:11:51.562352 24279 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:51.562479 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:51.562755 24279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:51.565068 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "f6f7305b4fb646c7b79872f6d2c7657d"
format_stamp: "Formatted at 2026-09-25 15:11:51 on dist-test-slave-64fx"
server_key: "7b03bba484dbbd2fbc59af5259c9a918"
server_key_iv: "1c60fb446faa2c5caec6205fe10f4df3"
server_key_version: "encryptionkey@0"
I20260925 15:11:51.565830 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "f6f7305b4fb646c7b79872f6d2c7657d"
format_stamp: "Formatted at 2026-09-25 15:11:51 on dist-test-slave-64fx"
server_key: "7b03bba484dbbd2fbc59af5259c9a918"
server_key_iv: "1c60fb446faa2c5caec6205fe10f4df3"
server_key_version: "encryptionkey@0"
I20260925 15:11:51.570081 24279 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 15:11:51.573282 24294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:51.574590 24279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:11:51.574762 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "f6f7305b4fb646c7b79872f6d2c7657d"
format_stamp: "Formatted at 2026-09-25 15:11:51 on dist-test-slave-64fx"
server_key: "7b03bba484dbbd2fbc59af5259c9a918"
server_key_iv: "1c60fb446faa2c5caec6205fe10f4df3"
server_key_version: "encryptionkey@0"
I20260925 15:11:51.574935 24279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:51.593779 24279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:51.597301 24279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:51.597615 24279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:51.598439 24279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:51.599613 24279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:51.599713 24279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:51.599864 24279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:51.599931 24279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:51.631595 24279 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:37487
I20260925 15:11:51.631644 24406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:37487 every 8 connection(s)
I20260925 15:11:51.632879 24279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:51.642930 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24279
I20260925 15:11:51.643110 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:51.643672 4919 external_mini_cluster.cc:1428] Setting key 5129918eaef197059673857873e38332
Sep 25 15:11:51 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349111, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.193@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:51.653970 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:51.635552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:38639 (local address 127.4.205.254:33969)
0925 15:11:51.635998 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:51.636005 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:51.637038 (+ 1033us) server_negotiation.cc:407] Connection header received
0925 15:11:51.638796 (+ 1758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:51.638800 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:51.638872 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:51.638966 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:51.640368 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.641086 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.642637 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.643305 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.643443 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:51.647073 (+ 3630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:51.647083 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:51.647091 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:51.647138 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:51.650749 (+ 3611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:51.651449 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:51.651458 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:51.651464 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:51.651619 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:51.652012 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:51.652020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:51.652026 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:51.652413 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:51.652562 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:51.653223 (+ 661us) server_negotiation.cc:299] Negotiation successful
0925 15:11:51.653469 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":207,"threads_started":1}
I20260925 15:11:51.654564 24410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:51.636093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33969 (local address 127.4.205.193:38639)
0925 15:11:51.636883 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:51.636929 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:51.638555 (+ 1626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:51.639161 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:51.639169 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:51.639654 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:51.640227 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:51.640243 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.641213 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.641216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:51.642490 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:51.642502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:51.643503 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:51.643508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:51.643701 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:51.644424 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:51.644450 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:51.646859 (+ 2409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:51.650876 (+ 4017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:51.650883 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:51.650896 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:51.651316 (+ 420us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:51.651725 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:51.651728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:51.651731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:51.651895 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:51.652524 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:51.652529 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:51.652842 (+ 313us) client_negotiation.cc:769] Sending connection context
0925 15:11:51.653070 (+ 228us) client_negotiation.cc:240] Negotiation successful
0925 15:11:51.653438 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":257,"threads_started":1}
I20260925 15:11:51.656126 24407 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33969
I20260925 15:11:51.656427 24407 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:51.657125 24407 heartbeater.cc:511] Master 127.4.205.254:33969 requested a full tablet report, sending...
I20260925 15:11:51.659158 24218 ts_manager.cc:194] Registered new tserver with Master: f6f7305b4fb646c7b79872f6d2c7657d (127.4.205.193:37487)
I20260925 15:11:51.660545 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.193@KRBTEST.COM'} at 127.4.205.193:38639
WARNING: no policy specified for kudu/127.4.205.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
Entry for principal kudu/127.4.205.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab.
I20260925 15:11:51.696798 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:52.001536 24413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:52.001856 24413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:52.002027 24413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:52.008955 24413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:52.009110 24413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:52.009287 24413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:52.018730 24413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.194.keytab
--principal=kudu/127.4.205.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:52.019999 24413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:52.021313 24413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:52.033993 24421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:52.034113 24419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:52.034149 24418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:52.034768 24413 server_base.cc:1034] running on GCE node
I20260925 15:11:52.035368 24413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:52.036187 24413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:52.037400 24413 hybrid_clock.cc:648] HybridClock initialized: now 1790349112037377 us; error 39 us; skew 500 ppm
I20260925 15:11:52.037756 24413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:52 dist-test-slave-64fx krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349112, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:52.040818 24413 init.cc:377] Logged in from keytab as kudu/127.4.205.194@KRBTEST.COM (short username kudu)
I20260925 15:11:52.042429 24413 webserver.cc:460] Webserver started at http://127.4.205.194:35043/ using document root <none> and password file <none>
I20260925 15:11:52.043171 24413 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:52.043286 24413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:52.043562 24413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:52.047129 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "e614249305b746419071eb67126fe5d7"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "c70f70cadda32f152a9f21dbbd61ad5e"
server_key_iv: "a71aac186ff8559ed12ef22c9ae9ec23"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.047987 24413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "e614249305b746419071eb67126fe5d7"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "c70f70cadda32f152a9f21dbbd61ad5e"
server_key_iv: "a71aac186ff8559ed12ef22c9ae9ec23"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.053777 24413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 15:11:52.057785 24428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.059181 24413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:11:52.059311 24413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "e614249305b746419071eb67126fe5d7"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "c70f70cadda32f152a9f21dbbd61ad5e"
server_key_iv: "a71aac186ff8559ed12ef22c9ae9ec23"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.059486 24413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:52.077194 24413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:52.080909 24413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:52.081143 24413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:52.082026 24413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:52.083231 24413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:52.083307 24413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.083384 24413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:52.083431 24413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.115393 24413 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:40605
I20260925 15:11:52.115490 24540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:40605 every 8 connection(s)
I20260925 15:11:52.116677 24413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:52.118026 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24413
I20260925 15:11:52.118225 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:52.118821 4919 external_mini_cluster.cc:1428] Setting key ed255ae0f789053f00b50bf1974b8774
Sep 25 15:11:52 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349112, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.194@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:52.139572 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.119447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:52001 (local address 127.4.205.254:33969)
0925 15:11:52.119646 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.119653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.120964 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 15:11:52.123906 (+ 2942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.123910 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.123972 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.124051 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:52.126044 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.126726 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.129158 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.129675 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.129776 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.133528 (+ 3752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:52.133537 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:52.133545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:52.133617 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:52.136484 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.137219 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.137227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.137236 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.137361 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.137835 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.137842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.137846 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.138129 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:52.138287 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.138980 (+ 693us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.139212 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:11:52.140564 24543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.120008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33969 (local address 127.4.205.194:52001)
0925 15:11:52.120801 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.120842 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.123588 (+ 2746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.124319 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.124333 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.125094 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:52.125869 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.125896 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.126881 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.126889 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.129027 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.129044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.129887 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.129896 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.130203 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.130943 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:52.130965 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:52.133301 (+ 2336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:52.136627 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:52.136637 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:52.136654 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:52.137089 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:52.137471 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:52.137477 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:52.137482 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:52.137725 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:52.138258 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:52.138265 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:52.138605 (+ 340us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.138885 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.139436 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":291,"threads_started":1}
I20260925 15:11:52.142056 24541 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33969
I20260925 15:11:52.142380 24541 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:52.143081 24541 heartbeater.cc:511] Master 127.4.205.254:33969 requested a full tablet report, sending...
I20260925 15:11:52.144541 24218 ts_manager.cc:194] Registered new tserver with Master: e614249305b746419071eb67126fe5d7 (127.4.205.194:40605)
WARNING: no policy specified for kudu/127.4.205.195@KRBTEST.COM; defaulting to no policy
I20260925 15:11:52.145392 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.194@KRBTEST.COM'} at 127.4.205.194:52001
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
Entry for principal kudu/127.4.205.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab.
I20260925 15:11:52.176783 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:36257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:52.479813 24546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:52.480098 24546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:52.480281 24546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:52.487202 24546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:52.487330 24546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:52.487504 24546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:52.496695 24546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:36257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.195.keytab
--principal=kudu/127.4.205.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:33969
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:52.497851 24546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:52.499068 24546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:52.511358 24551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:52.511278 24552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:52.511324 24554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:52.512080 24546 server_base.cc:1034] running on GCE node
I20260925 15:11:52.512611 24546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:52.513501 24546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:52.514761 24546 hybrid_clock.cc:648] HybridClock initialized: now 1790349112514716 us; error 35 us; skew 500 ppm
I20260925 15:11:52.515064 24546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:52 dist-test-slave-64fx krb5kdc[24173](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349112, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:52.517879 24546 init.cc:377] Logged in from keytab as kudu/127.4.205.195@KRBTEST.COM (short username kudu)
I20260925 15:11:52.519145 24546 webserver.cc:460] Webserver started at http://127.4.205.195:43945/ using document root <none> and password file <none>
I20260925 15:11:52.519693 24546 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:52.519784 24546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:52.520009 24546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:52.522706 24546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "f4eed99cdc914ec09568ee6ebdec6ca4"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "79e26a65cfd41d5f0bd8b38b433d578e"
server_key_iv: "c68231f93a28dee620f3f193fdee9cc1"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.523363 24546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "f4eed99cdc914ec09568ee6ebdec6ca4"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "79e26a65cfd41d5f0bd8b38b433d578e"
server_key_iv: "c68231f93a28dee620f3f193fdee9cc1"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.527664 24546 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:11:52.530795 24561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.531927 24546 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:11:52.532104 24546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "f4eed99cdc914ec09568ee6ebdec6ca4"
format_stamp: "Formatted at 2026-09-25 15:11:52 on dist-test-slave-64fx"
server_key: "79e26a65cfd41d5f0bd8b38b433d578e"
server_key_iv: "c68231f93a28dee620f3f193fdee9cc1"
server_key_version: "encryptionkey@0"
I20260925 15:11:52.532282 24546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:52.550726 24546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:52.554083 24546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:52.554345 24546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:52.555162 24546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:52.556396 24546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:52.556463 24546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.556620 24546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:52.556668 24546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:52.587880 24546 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:45847
I20260925 15:11:52.587958 24673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:45847 every 8 connection(s)
I20260925 15:11:52.589072 24546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:52.598582 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24546
I20260925 15:11:52.598759 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:52.599282 4919 external_mini_cluster.cc:1428] Setting key 53c8404fe5fe377521f299a169177da4
Sep 25 15:11:52 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349112, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.195@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:11:52.606945 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.591472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:57043 (local address 127.4.205.254:33969)
0925 15:11:52.591644 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.591648 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.592701 (+ 1053us) server_negotiation.cc:407] Connection header received
0925 15:11:52.594440 (+ 1739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.594444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.594503 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.594582 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:52.595920 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.596588 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.597968 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.598349 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.598440 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.602461 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:52.602465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:52.602468 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:52.602494 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:52.604587 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.605177 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.605180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.605182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.605295 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.605713 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.605716 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.605718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.605963 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:52.606082 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.606575 (+ 493us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.606716 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:11:52.607662 24676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.591889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:33969 (local address 127.4.205.195:57043)
0925 15:11:52.592562 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.592599 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.594172 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.594769 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.594776 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.595263 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:52.595786 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.595802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.596718 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.596721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.597848 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.597859 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.598557 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.598566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.598846 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.599446 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:11:52.599480 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:52.602238 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:52.604694 (+ 2456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:52.604699 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:52.604708 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:52.605059 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:52.605376 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:11:52.605379 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:11:52.605381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:11:52.605548 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:11:52.606076 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:52.606081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:52.606315 (+ 234us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.606497 (+ 182us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.606779 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":259,"threads_started":1}
I20260925 15:11:52.608850 24674 heartbeater.cc:347] Connected to a master server at 127.4.205.254:33969
I20260925 15:11:52.609069 24674 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:52.609709 24674 heartbeater.cc:511] Master 127.4.205.254:33969 requested a full tablet report, sending...
I20260925 15:11:52.610997 24218 ts_manager.cc:194] Registered new tserver with Master: f4eed99cdc914ec09568ee6ebdec6ca4 (127.4.205.195:45847)
I20260925 15:11:52.611773 24218 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.4.205.195@KRBTEST.COM'} at 127.4.205.195:57043
I20260925 15:11:52.614358 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:52.629732 24409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.618456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.4.205.254:33969)
0925 15:11:52.618597 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.618601 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.619017 (+ 416us) server_negotiation.cc:407] Connection header received
0925 15:11:52.619131 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.619135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.619197 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.619272 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:52.620574 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.621257 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.622538 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.622917 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.622999 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.625030 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:52.625034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:52.625037 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:52.625064 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:52.627526 (+ 2462us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.628024 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.628028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.628030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.628143 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.628474 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.628476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.628478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.628717 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:52.628892 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.629236 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.629374 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:11:52.633162 24218 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:52.639828 24218 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:52.663334 24407 heartbeater.cc:503] Master 127.4.205.254:33969 was elected leader, sending a full tablet report...
I20260925 15:11:52.675850 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.651377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51868 (local address 127.4.205.195:45847)
0925 15:11:52.666618 (+ 15241us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.666630 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.667662 (+ 1032us) server_negotiation.cc:407] Connection header received
0925 15:11:52.667800 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.667811 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.668150 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.668371 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:52.669651 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.671325 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.673702 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.674734 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.674856 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.675091 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.675351 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.675508 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15021,"thread_start_us":15261,"threads_started":1}
I20260925 15:11:52.675935 24690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.650921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45847 (local address 127.0.0.1:51868)
0925 15:11:52.667321 (+ 16400us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.667350 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.667501 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.668719 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.668730 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.668770 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:52.669282 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.669297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.671462 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.671468 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.673501 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.673516 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.674919 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.674925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.675116 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.675166 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.675302 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.675481 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16251,"mutex_wait_us":51,"thread_start_us":18128,"threads_started":1}
I20260925 15:11:52.676621 24686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.650768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:37487 (local address 127.0.0.1:34198)
0925 15:11:52.666576 (+ 15808us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.666631 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.666837 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.669260 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.669264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.669286 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:52.669806 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.669818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.671693 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.671699 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.673807 (+ 2108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.673820 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.675803 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.675810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.676008 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.676054 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.676134 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.676261 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15581,"mutex_wait_us":55,"thread_start_us":15432,"threads_started":1}
I20260925 15:11:52.677419 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.650913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34198 (local address 127.4.205.193:37487)
0925 15:11:52.668235 (+ 17322us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.668245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.668270 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:11:52.668351 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.668357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.668560 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.668739 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:52.669980 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.671566 (+ 1586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.673964 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.675345 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.675459 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.675664 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.676950 (+ 1286us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.677123 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17149,"thread_start_us":16965,"threads_started":1}
I20260925 15:11:52.677686 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.653607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.4.205.194:40605)
0925 15:11:52.658569 (+ 4962us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.658582 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.669459 (+ 10877us) server_negotiation.cc:407] Connection header received
0925 15:11:52.669823 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.669832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.670036 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.670205 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:52.671004 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.672378 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.674724 (+ 2346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.676082 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.676202 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.676399 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.677227 (+ 828us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.677383 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4754,"thread_start_us":4622,"threads_started":1}
I20260925 15:11:52.678109 24688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.653033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:40605 (local address 127.0.0.1:55854)
0925 15:11:52.669340 (+ 16307us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.669370 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.669587 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.670458 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.670462 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.670484 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:52.670840 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.670850 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.672549 (+ 1699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.672560 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.674554 (+ 1994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.674568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.676546 (+ 1978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.676553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.676726 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.676757 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.676859 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.676972 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16164,"mutex_wait_us":6,"thread_start_us":13987,"threads_started":1}
I20260925 15:11:52.679822 24609 tablet_service.cc:1467] Processing CreateTablet for tablet b48e85aa271b495597738a307157f8f6 (DEFAULT_TABLE table=test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:52.681021 24609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48e85aa271b495597738a307157f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:52.680770 24342 tablet_service.cc:1467] Processing CreateTablet for tablet b48e85aa271b495597738a307157f8f6 (DEFAULT_TABLE table=test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:52.681473 24476 tablet_service.cc:1467] Processing CreateTablet for tablet b48e85aa271b495597738a307157f8f6 (DEFAULT_TABLE table=test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:52.682196 24342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48e85aa271b495597738a307157f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:52.682780 24476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b48e85aa271b495597738a307157f8f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:52.691799 24692 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Bootstrap starting.
I20260925 15:11:52.695166 24692 tablet_bootstrap.cc:654] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:52.696218 24692 log.cc:824] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:52.698168 24694 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Bootstrap starting.
I20260925 15:11:52.699218 24692 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: No bootstrap required, opened a new log
I20260925 15:11:52.699483 24692 ts_tablet_manager.cc:1397] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260925 15:11:52.697453 24693 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Bootstrap starting.
I20260925 15:11:52.701380 24694 tablet_bootstrap.cc:654] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:52.702632 24694 log.cc:824] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:52.704631 24693 tablet_bootstrap.cc:654] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:52.705591 24693 log.cc:824] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:52.706229 24694 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: No bootstrap required, opened a new log
I20260925 15:11:52.706497 24694 ts_tablet_manager.cc:1397] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 15:11:52.709602 24692 raft_consensus.cc:348] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.710105 24692 raft_consensus.cc:374] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:52.710206 24692 raft_consensus.cc:729] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4eed99cdc914ec09568ee6ebdec6ca4, State: Initialized, Role: FOLLOWER
I20260925 15:11:52.710780 24692 consensus_queue.cc:260] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.711838 24674 heartbeater.cc:503] Master 127.4.205.254:33969 was elected leader, sending a full tablet report...
I20260925 15:11:52.712034 24693 tablet_bootstrap.cc:492] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: No bootstrap required, opened a new log
I20260925 15:11:52.712271 24693 ts_tablet_manager.cc:1397] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.003s
I20260925 15:11:52.713723 24692 ts_tablet_manager.cc:1428] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Time spent starting tablet: real 0.014s user 0.010s sys 0.005s
I20260925 15:11:52.716624 24694 raft_consensus.cc:348] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.717051 24694 raft_consensus.cc:374] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:52.717171 24694 raft_consensus.cc:729] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e614249305b746419071eb67126fe5d7, State: Initialized, Role: FOLLOWER
I20260925 15:11:52.717787 24694 consensus_queue.cc:260] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.718758 24541 heartbeater.cc:503] Master 127.4.205.254:33969 was elected leader, sending a full tablet report...
I20260925 15:11:52.719249 24694 ts_tablet_manager.cc:1428] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260925 15:11:52.720935 24693 raft_consensus.cc:348] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.721242 24693 raft_consensus.cc:374] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:52.721385 24693 raft_consensus.cc:729] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6f7305b4fb646c7b79872f6d2c7657d, State: Initialized, Role: FOLLOWER
I20260925 15:11:52.721997 24693 consensus_queue.cc:260] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.723333 24693 ts_tablet_manager.cc:1428] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260925 15:11:52.787367 24699 raft_consensus.cc:482] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:52.787606 24699 raft_consensus.cc:504] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.789055 24699 leader_election.cc:288] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f6f7305b4fb646c7b79872f6d2c7657d (127.4.205.193:37487), f4eed99cdc914ec09568ee6ebdec6ca4 (127.4.205.195:45847)
I20260925 15:11:52.795781 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.789605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53727 (local address 127.4.205.195:45847)
0925 15:11:52.789823 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.789831 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.790523 (+ 692us) server_negotiation.cc:407] Connection header received
0925 15:11:52.790661 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.790665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.790754 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.790845 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:52.791465 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.792231 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.794030 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.794845 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.794961 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.795064 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.795404 (+ 340us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.795531 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:11:52.795812 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.789393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53577 (local address 127.4.205.193:37487)
0925 15:11:52.789605 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.789612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.790391 (+ 779us) server_negotiation.cc:407] Connection header received
0925 15:11:52.790543 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.790548 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.790633 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.790709 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:52.791330 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.792068 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.794052 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.794922 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.795070 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.795142 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.795466 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.795601 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:11:52.795864 24702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.789413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:45847 (local address 127.4.205.194:53727)
0925 15:11:52.790407 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.790423 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.790546 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.790884 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.790887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.790916 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:52.791313 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.791329 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.792371 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.792378 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.793878 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.793888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.795045 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.795052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.795236 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.795273 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.795342 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.795462 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":4,"thread_start_us":502,"threads_started":1}
I20260925 15:11:52.795944 24701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.789393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:37487 (local address 127.4.205.194:53577)
0925 15:11:52.790273 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:52.790287 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:52.790414 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:52.790747 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:52.790752 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:52.790776 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:52.791160 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.791169 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.792195 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.792202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.793887 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:52.793899 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.795140 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.795144 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:52.795248 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.795312 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 15:11:52.795422 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:11:52.795542 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"mutex_wait_us":73,"thread_start_us":652,"threads_started":1}
I20260925 15:11:52.796901 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48e85aa271b495597738a307157f8f6" candidate_uuid: "e614249305b746419071eb67126fe5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" is_pre_election: true
I20260925 15:11:52.797199 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48e85aa271b495597738a307157f8f6" candidate_uuid: "e614249305b746419071eb67126fe5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" is_pre_election: true
I20260925 15:11:52.797453 24362 raft_consensus.cc:2454] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e614249305b746419071eb67126fe5d7 in term 0.
I20260925 15:11:52.797731 24629 raft_consensus.cc:2454] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e614249305b746419071eb67126fe5d7 in term 0.
I20260925 15:11:52.798205 24431 leader_election.cc:302] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e614249305b746419071eb67126fe5d7, f6f7305b4fb646c7b79872f6d2c7657d; no voters:
I20260925 15:11:52.798556 24699 raft_consensus.cc:2781] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:52.798704 24699 raft_consensus.cc:482] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:52.798804 24699 raft_consensus.cc:3037] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:52.800990 24699 raft_consensus.cc:504] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.801659 24699 leader_election.cc:288] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [CANDIDATE]: Term 1 election: Requested vote from peers f6f7305b4fb646c7b79872f6d2c7657d (127.4.205.193:37487), f4eed99cdc914ec09568ee6ebdec6ca4 (127.4.205.195:45847)
I20260925 15:11:52.802006 24362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48e85aa271b495597738a307157f8f6" candidate_uuid: "e614249305b746419071eb67126fe5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6f7305b4fb646c7b79872f6d2c7657d"
I20260925 15:11:52.802098 24629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b48e85aa271b495597738a307157f8f6" candidate_uuid: "e614249305b746419071eb67126fe5d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4"
I20260925 15:11:52.802201 24362 raft_consensus.cc:3037] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:52.802268 24629 raft_consensus.cc:3037] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:52.805172 24362 raft_consensus.cc:2454] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e614249305b746419071eb67126fe5d7 in term 1.
I20260925 15:11:52.805169 24629 raft_consensus.cc:2454] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e614249305b746419071eb67126fe5d7 in term 1.
I20260925 15:11:52.805667 24431 leader_election.cc:302] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: e614249305b746419071eb67126fe5d7, f4eed99cdc914ec09568ee6ebdec6ca4, f6f7305b4fb646c7b79872f6d2c7657d; no voters:
I20260925 15:11:52.805992 24699 raft_consensus.cc:2781] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:52.806418 24699 raft_consensus.cc:686] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 1 LEADER]: Becoming Leader. State: Replica: e614249305b746419071eb67126fe5d7, State: Running, Role: LEADER
I20260925 15:11:52.806859 24699 consensus_queue.cc:237] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } }
I20260925 15:11:52.810879 24218 catalog_manager.cc:5495] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to e614249305b746419071eb67126fe5d7 (127.4.205.194). New cstate: current_term: 1 leader_uuid: "e614249305b746419071eb67126fe5d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e614249305b746419071eb67126fe5d7" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 40605 } health_report { overall_health: HEALTHY } } }
I20260925 15:11:52.825747 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.821063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55858 (local address 127.4.205.194:40605)
0925 15:11:52.821288 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.821294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.821364 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:11:52.821515 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.821518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.821651 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.821770 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:52.822082 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.822783 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.823968 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.824319 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.824397 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.824778 (+ 381us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:52.825198 (+ 420us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:52.825301 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.825366 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.825421 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:11:52.833153 24629 raft_consensus.cc:1261] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 1 FOLLOWER]: Refusing update from remote peer e614249305b746419071eb67126fe5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:52.833194 24362 raft_consensus.cc:1261] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 1 FOLLOWER]: Refusing update from remote peer e614249305b746419071eb67126fe5d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:52.834111 24703 consensus_queue.cc:1035] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6f7305b4fb646c7b79872f6d2c7657d" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 37487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:52.834407 24699 consensus_queue.cc:1035] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f4eed99cdc914ec09568ee6ebdec6ca4" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 45847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:11:52.840565 24675 tablet.cc:2369] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:52.851848 24707 mvcc.cc:205] Tried to move back new op lower bound from 7333269966155628544 to 7333269966060445696. Current Snapshot: MvccSnapshot[applied={T|T < 7333269966155628544}]
I20260925 15:11:52.856393 24708 mvcc.cc:205] Tried to move back new op lower bound from 7333269966155628544 to 7333269966060445696. Current Snapshot: MvccSnapshot[applied={T|T < 7333269966155628544}]
I20260925 15:11:52.871436 24218 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:52.871718 24218 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60310:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:52.876176 24218 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3ed2beecd2e4e15b4d9948b7cedc604: Sending DeleteTablet for 3 replicas of tablet b48e85aa271b495597738a307157f8f6
I20260925 15:11:52.877290 24342 tablet_service.cc:1514] Processing DeleteTablet for tablet b48e85aa271b495597738a307157f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:52 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:34198
I20260925 15:11:52.877446 24609 tablet_service.cc:1514] Processing DeleteTablet for tablet b48e85aa271b495597738a307157f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:52 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:51868
I20260925 15:11:52.878795 24476 tablet_service.cc:1514] Processing DeleteTablet for tablet b48e85aa271b495597738a307157f8f6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:52 UTC) from {username='kudu', principal='kudu/127.4.205.254@KRBTEST.COM'} at 127.0.0.1:55854
I20260925 15:11:52.887230 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24279
I20260925 15:11:52.902833 24717 tablet_replica.cc:330] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: stopping tablet replica
I20260925 15:11:52.903373 24717 raft_consensus.cc:2229] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:52.904080 24717 raft_consensus.cc:2258] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:52.904855 24716 tablet_replica.cc:330] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: stopping tablet replica
I20260925 15:11:52.905280 24716 raft_consensus.cc:2229] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:52.905822 24716 raft_consensus.cc:2258] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:52.908607 24717 ts_tablet_manager.cc:1905] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:52.910122 24715 tablet_replica.cc:330] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: stopping tablet replica
I20260925 15:11:52.910475 24715 raft_consensus.cc:2229] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:52.910892 24715 raft_consensus.cc:2258] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:52.911652 24716 ts_tablet_manager.cc:1905] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:52.912760 24715 ts_tablet_manager.cc:1905] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 15:11:52 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.193@KRBTEST.COM
I20260925 15:11:52.919350 24715 ts_tablet_manager.cc:1918] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:52.919515 24715 log.cc:1196] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/b48e85aa271b495597738a307157f8f6
I20260925 15:11:52.920148 24715 ts_tablet_manager.cc:1939] T b48e85aa271b495597738a307157f8f6 P f6f7305b4fb646c7b79872f6d2c7657d: Deleting consensus metadata
I20260925 15:11:52.922053 24716 ts_tablet_manager.cc:1918] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:52.922202 24716 log.cc:1196] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/b48e85aa271b495597738a307157f8f6
I20260925 15:11:52.922288 24204 catalog_manager.cc:4841] TS f6f7305b4fb646c7b79872f6d2c7657d (127.4.205.193:37487): tablet b48e85aa271b495597738a307157f8f6 (table test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]) successfully deleted
I20260925 15:11:52.922758 24716 ts_tablet_manager.cc:1939] T b48e85aa271b495597738a307157f8f6 P f4eed99cdc914ec09568ee6ebdec6ca4: Deleting consensus metadata
I20260925 15:11:52.922994 24717 ts_tablet_manager.cc:1918] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:52.924306 24202 catalog_manager.cc:4841] TS f4eed99cdc914ec09568ee6ebdec6ca4 (127.4.205.195:45847): tablet b48e85aa271b495597738a307157f8f6 (table test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]) successfully deleted
I20260925 15:11:52.925177 24717 log.cc:1196] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/b48e85aa271b495597738a307157f8f6
I20260925 15:11:52.925802 24717 ts_tablet_manager.cc:1939] T b48e85aa271b495597738a307157f8f6 P e614249305b746419071eb67126fe5d7: Deleting consensus metadata
I20260925 15:11:52.927421 24203 catalog_manager.cc:4841] TS e614249305b746419071eb67126fe5d7 (127.4.205.194:40605): tablet b48e85aa271b495597738a307157f8f6 (table test-table [id=fb3aa2c3d21c47b3adaa4f3b6ebaac05]) successfully deleted
I20260925 15:11:52.929054 24687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:52.888870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34202 (local address 127.4.205.193:37487)
0925 15:11:52.889043 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:52.889049 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:52.902015 (+ 12966us) server_negotiation.cc:407] Connection header received
0925 15:11:52.902503 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:52.902509 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:52.902591 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:52.902662 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:52.904597 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.905781 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.908419 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:52.909076 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:52.909166 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:52.920869 (+ 11703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:52.920887 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:52.920909 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:52.920957 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:52.924916 (+ 3959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.925994 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.926006 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.926012 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.926193 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:52.926702 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:52.926709 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:52.926715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:52.927255 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:52.927598 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:52.928274 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 15:11:52.928509 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:11:52.929915 24402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:53.017104 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24279
I20260925 15:11:53.039547 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24413
Sep 25 15:11:53 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.194@KRBTEST.COM
I20260925 15:11:53.058642 24691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:53.040229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55872 (local address 127.4.205.194:40605)
0925 15:11:53.040413 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:53.040419 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:53.040713 (+ 294us) server_negotiation.cc:407] Connection header received
0925 15:11:53.040908 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:53.040915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:53.041014 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:53.041148 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:53.042785 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.044052 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.045859 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.046496 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.046611 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:53.051849 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:53.051865 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:53.051885 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:53.051933 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:53.055759 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:53.056510 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:53.056520 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:53.056523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:53.056684 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:53.057081 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:53.057086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:53.057089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:53.057500 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:53.057824 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:53.058055 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 15:11:53.058252 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:11:53.059322 24536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:53.168936 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24413
I20260925 15:11:53.188414 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24546
Sep 25 15:11:53 dist-test-slave-64fx krb5kdc[24173](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349110, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.195@KRBTEST.COM
I20260925 15:11:53.202528 24689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:53.189119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.4.205.195:45847)
0925 15:11:53.189271 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:53.189275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:53.189292 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:11:53.189455 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:53.189459 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:53.189530 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:53.189746 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:53.190914 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.191641 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.192836 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.193206 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.193290 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:53.197543 (+ 4253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:53.197599 (+ 56us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:53.197620 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:53.197650 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:53.200006 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:53.200671 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:53.200678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:53.200680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:53.200810 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:53.201132 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:53.201135 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:53.201137 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:53.201520 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:53.201808 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:53.202025 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 15:11:53.202190 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:11:53.203212 24669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:53.290674 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24546
I20260925 15:11:53.307410 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24186
I20260925 15:11:53.308475 24248 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:53.416150 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24186
2026-09-25T15:11:53Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4094 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:11:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:53Z Disabled control of system clock
I20260925 15:11:53.486119 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41637
--webserver_interface=127.4.205.254
--webserver_port=0
--builtin_ntp_servers=127.4.205.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41637
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:53.789834 24726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:53.790076 24726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:53.790186 24726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:53.796744 24726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:53.796844 24726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:53.796885 24726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:53.796948 24726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:53.796989 24726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:53.805344 24726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41637
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:53.806363 24726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:53.807893 24726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:53.818521 24731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:53.818547 24732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:53.818523 24734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:53.819418 24726 server_base.cc:1034] running on GCE node
I20260925 15:11:53.820083 24726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:53.821355 24726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:53.822590 24726 hybrid_clock.cc:648] HybridClock initialized: now 1790349113822517 us; error 83 us; skew 500 ppm
I20260925 15:11:53.822928 24726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:53.824924 24726 webserver.cc:460] Webserver started at http://127.4.205.254:40831/ using document root <none> and password file <none>
I20260925 15:11:53.825651 24726 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:53.825779 24726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:53.826105 24726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:53.828648 24726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "861b5c4dff2b47c28eb5a2969396ba4f"
format_stamp: "Formatted at 2026-09-25 15:11:53 on dist-test-slave-64fx"
server_key: "e95c16e7ad9d6d8b3af3ab32b8231465"
server_key_iv: "d6ce3ec4558966a5ce864b0dc72a04bc"
server_key_version: "encryptionkey@0"
I20260925 15:11:53.829386 24726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "861b5c4dff2b47c28eb5a2969396ba4f"
format_stamp: "Formatted at 2026-09-25 15:11:53 on dist-test-slave-64fx"
server_key: "e95c16e7ad9d6d8b3af3ab32b8231465"
server_key_iv: "d6ce3ec4558966a5ce864b0dc72a04bc"
server_key_version: "encryptionkey@0"
I20260925 15:11:53.834141 24726 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:11:53.837275 24740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:53.838591 24726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:11:53.838773 24726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "861b5c4dff2b47c28eb5a2969396ba4f"
format_stamp: "Formatted at 2026-09-25 15:11:53 on dist-test-slave-64fx"
server_key: "e95c16e7ad9d6d8b3af3ab32b8231465"
server_key_iv: "d6ce3ec4558966a5ce864b0dc72a04bc"
server_key_version: "encryptionkey@0"
I20260925 15:11:53.838984 24726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:53.854769 24726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:53.855504 24726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:53.855770 24726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:53.872752 24726 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41637
I20260925 15:11:53.872774 24791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41637 every 8 connection(s)
I20260925 15:11:53.874014 24726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:53.877481 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:53.884362 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24726
I20260925 15:11:53.884531 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:53.885051 4919 external_mini_cluster.cc:1428] Setting key c3763ccd87b747a110d9811892093e4f
I20260925 15:11:53.887854 24792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: Bootstrap starting.
I20260925 15:11:53.890841 24792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:53.891870 24792 log.cc:824] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:53.894424 24792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: No bootstrap required, opened a new log
I20260925 15:11:53.897673 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:53.886854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44762 (local address 127.4.205.254:41637)
0925 15:11:53.887735 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:53.887745 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:53.887783 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:11:53.889492 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:53.889512 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:53.889985 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:53.890328 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:53.890846 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.892208 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.893490 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:53.894183 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:53.894370 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:53.895506 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:53.895513 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:53.895526 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:53.896014 (+ 488us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:53.896090 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:53.896111 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:53.896235 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:53.896453 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:11:53.896739 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
I20260925 15:11:53.902369 24792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } }
I20260925 15:11:53.902771 24792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:53.902871 24792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 861b5c4dff2b47c28eb5a2969396ba4f, State: Initialized, Role: FOLLOWER
I20260925 15:11:53.903373 24792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } }
I20260925 15:11:53.903600 24792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:53.903743 24792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:53.903914 24792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:53.906313 24792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } }
I20260925 15:11:53.906786 24792 leader_election.cc:302] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 861b5c4dff2b47c28eb5a2969396ba4f; no voters:
I20260925 15:11:53.907279 24792 leader_election.cc:288] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:53.907439 24797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:53.908108 24797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [term 1 LEADER]: Becoming Leader. State: Replica: 861b5c4dff2b47c28eb5a2969396ba4f, State: Running, Role: LEADER
I20260925 15:11:53.908571 24797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } }
I20260925 15:11:53.909200 24792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:53.911194 24799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 861b5c4dff2b47c28eb5a2969396ba4f. Latest consensus state: current_term: 1 leader_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } } }
I20260925 15:11:53.911610 24799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:53.912055 24798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "861b5c4dff2b47c28eb5a2969396ba4f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41637 } } }
I20260925 15:11:53.912353 24798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:53.918902 24805 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:53.923596 24805 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:53.931672 24805 catalog_manager.cc:1347] Generated new cluster ID: 996dcc202ba64ed6b7edef3bddf81be4
I20260925 15:11:53.931782 24805 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:53.941063 24805 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:53.942764 24805 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:53.951296 24805 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: Generated new TSK 0
I20260925 15:11:53.952208 24805 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:11:53.965055 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.193:0
--local_ip_for_outbound_sockets=127.4.205.193
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--builtin_ntp_servers=127.4.205.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:11:54.282172 24816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:54.282488 24816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:54.282647 24816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:54.289729 24816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:54.289851 24816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:54.290020 24816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.193
I20260925 15:11:54.298709 24816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.4.205.193
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.193
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:54.299855 24816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:54.301522 24816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:54.314589 24821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:54.314657 24824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:54.314700 24822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:54.315431 24816 server_base.cc:1034] running on GCE node
I20260925 15:11:54.316005 24816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:54.316901 24816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:54.318107 24816 hybrid_clock.cc:648] HybridClock initialized: now 1790349114318078 us; error 47 us; skew 500 ppm
I20260925 15:11:54.318377 24816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:54.320257 24816 webserver.cc:460] Webserver started at http://127.4.205.193:35825/ using document root <none> and password file <none>
I20260925 15:11:54.320904 24816 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:54.321056 24816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:54.321367 24816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:54.323966 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data/instance:
uuid: "769ababfd5944e488c79713cb00a4ccb"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "b7dc0332cfd7e21e5eab79b28f629bc6"
server_key_iv: "a073b548e8bb27e611af94124a021ac2"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.324704 24816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance:
uuid: "769ababfd5944e488c79713cb00a4ccb"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "b7dc0332cfd7e21e5eab79b28f629bc6"
server_key_iv: "a073b548e8bb27e611af94124a021ac2"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.329139 24816 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260925 15:11:54.332324 24830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.333495 24816 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:11:54.333690 24816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal
uuid: "769ababfd5944e488c79713cb00a4ccb"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "b7dc0332cfd7e21e5eab79b28f629bc6"
server_key_iv: "a073b548e8bb27e611af94124a021ac2"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.333902 24816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:54.343775 24816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:54.344390 24816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:54.344672 24816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:54.345475 24816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:54.346915 24816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:54.347035 24816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.347172 24816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:54.347257 24816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.377794 24816 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.193:34509
I20260925 15:11:54.377864 24942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.193:34509 every 8 connection(s)
I20260925 15:11:54.379011 24816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/data/info.pb
I20260925 15:11:54.386687 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24816
I20260925 15:11:54.386861 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal/instance
I20260925 15:11:54.387300 4919 external_mini_cluster.cc:1428] Setting key 9df62918e5fdc83474815398a548b1ec
I20260925 15:11:54.390242 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.194:0
--local_ip_for_outbound_sockets=127.4.205.194
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--builtin_ntp_servers=127.4.205.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:54.391362 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:54.381447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:49091 (local address 127.4.205.254:41637)
0925 15:11:54.381676 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:54.381680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:54.382871 (+ 1191us) server_negotiation.cc:407] Connection header received
0925 15:11:54.384584 (+ 1713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:54.384591 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:54.384675 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:54.384785 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:54.385822 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.386650 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.388330 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.388967 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.389092 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:54.390503 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:54.390510 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:54.390517 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:54.390539 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:54.390575 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:54.390590 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:54.390672 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:54.390882 (+ 210us) server_negotiation.cc:299] Negotiation successful
0925 15:11:54.390980 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:11:54.392014 24945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:54.381892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41637 (local address 127.4.205.193:49091)
0925 15:11:54.382730 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:54.382764 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:54.384288 (+ 1524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:54.385000 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:54.385008 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:54.385091 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:54.385665 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:54.385682 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.386847 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.386851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:54.388210 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:54.388221 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.389214 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.389223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:54.389475 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:54.390141 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:54.390147 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:54.390192 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:54.390275 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:54.390684 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:54.390693 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:54.390699 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 15:11:54.390824 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:11:54.391072 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":256,"threads_started":1}
I20260925 15:11:54.393496 24943 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41637
I20260925 15:11:54.393793 24943 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:54.394606 24943 heartbeater.cc:511] Master 127.4.205.254:41637 requested a full tablet report, sending...
I20260925 15:11:54.397004 24757 ts_manager.cc:194] Registered new tserver with Master: 769ababfd5944e488c79713cb00a4ccb (127.4.205.193:34509)
I20260925 15:11:54.399180 24757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.193:49091
W20260925 15:11:54.707370 24946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:54.707644 24946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:54.707785 24946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:54.715029 24946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:54.715142 24946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:54.715320 24946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.194
I20260925 15:11:54.723702 24946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.4.205.194
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.194
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:54.724804 24946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:54.726491 24946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:54.738368 24952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:54.738435 24954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:54.738461 24951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:54.739172 24946 server_base.cc:1034] running on GCE node
I20260925 15:11:54.739711 24946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:54.740561 24946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:54.741736 24946 hybrid_clock.cc:648] HybridClock initialized: now 1790349114741701 us; error 32 us; skew 500 ppm
I20260925 15:11:54.742043 24946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:54.744233 24946 webserver.cc:460] Webserver started at http://127.4.205.194:38721/ using document root <none> and password file <none>
I20260925 15:11:54.744827 24946 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:54.744931 24946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:54.745230 24946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:54.747761 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data/instance:
uuid: "e32b6fadbb514d21bf233a0a5f718cab"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "7101d18b64bcc5200067cc36fe31cd89"
server_key_iv: "df585db05be93a95d09baaac8c988014"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.748489 24946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance:
uuid: "e32b6fadbb514d21bf233a0a5f718cab"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "7101d18b64bcc5200067cc36fe31cd89"
server_key_iv: "df585db05be93a95d09baaac8c988014"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.753203 24946 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 15:11:54.756417 24960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.757544 24946 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260925 15:11:54.757759 24946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal
uuid: "e32b6fadbb514d21bf233a0a5f718cab"
format_stamp: "Formatted at 2026-09-25 15:11:54 on dist-test-slave-64fx"
server_key: "7101d18b64bcc5200067cc36fe31cd89"
server_key_iv: "df585db05be93a95d09baaac8c988014"
server_key_version: "encryptionkey@0"
I20260925 15:11:54.757963 24946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:54.785684 24946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:54.786423 24946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:54.786696 24946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:54.787525 24946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:54.788832 24946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:54.788944 24946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.789072 24946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:54.789153 24946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:54.819965 24946 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.194:44833
I20260925 15:11:54.820015 25072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.194:44833 every 8 connection(s)
I20260925 15:11:54.821215 24946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/data/info.pb
I20260925 15:11:54.823259 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 24946
I20260925 15:11:54.823457 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal/instance
I20260925 15:11:54.823984 4919 external_mini_cluster.cc:1428] Setting key 5b2bfba14e96ef0a2a4de61cd41be7a3
I20260925 15:11:54.826432 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.205.195:0
--local_ip_for_outbound_sockets=127.4.205.195
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--builtin_ntp_servers=127.4.205.212:44929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:11:54.834111 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:54.823832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.194:53983 (local address 127.4.205.254:41637)
0925 15:11:54.824052 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:54.824058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:54.825543 (+ 1485us) server_negotiation.cc:407] Connection header received
0925 15:11:54.827335 (+ 1792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:54.827343 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:54.827428 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:54.827534 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:54.828457 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.829504 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.831082 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.831817 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.831954 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:54.833181 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:54.833188 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:54.833195 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:54.833217 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:54.833267 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:54.833283 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:54.833367 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:54.833569 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 15:11:54.833698 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:11:54.835129 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:54.824410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41637 (local address 127.4.205.194:53983)
0925 15:11:54.825394 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:54.825442 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:54.827090 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:54.827714 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:54.827721 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:54.827799 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:54.828303 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:54.828321 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.829665 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.829670 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:54.830938 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:54.830949 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:54.832012 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:54.832017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:54.832210 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:54.832883 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:54.832885 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:54.832926 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:54.832994 (+ 68us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:54.833370 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:54.833377 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:54.833382 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:11:54.833510 (+ 128us) client_negotiation.cc:240] Negotiation successful
0925 15:11:54.833889 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":317,"threads_started":1}
I20260925 15:11:54.836374 25073 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41637
I20260925 15:11:54.836658 25073 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:54.837387 25073 heartbeater.cc:511] Master 127.4.205.254:41637 requested a full tablet report, sending...
I20260925 15:11:54.838862 24757 ts_manager.cc:194] Registered new tserver with Master: e32b6fadbb514d21bf233a0a5f718cab (127.4.205.194:44833)
I20260925 15:11:54.839689 24757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.194:53983
W20260925 15:11:55.150375 25076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:55.150662 25076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:55.150817 25076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:55.157796 25076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:55.157913 25076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:55.158084 25076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.205.195
I20260925 15:11:55.167094 25076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.205.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.4.205.195
--webserver_port=0
--tserver_master_addrs=127.4.205.254:41637
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.205.195
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:55.168300 25076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:55.170046 25076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:55.182499 25081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:55.182471 25082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:55.182526 25084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:55.183532 25076 server_base.cc:1034] running on GCE node
I20260925 15:11:55.184262 25076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:55.185441 25076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:55.186681 25076 hybrid_clock.cc:648] HybridClock initialized: now 1790349115186611 us; error 71 us; skew 500 ppm
I20260925 15:11:55.187006 25076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:11:55.189067 25076 webserver.cc:460] Webserver started at http://127.4.205.195:36495/ using document root <none> and password file <none>
I20260925 15:11:55.189817 25076 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:55.189929 25076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:55.190256 25076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:55.192965 25076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data/instance:
uuid: "2b8d22acf6684427a6c0da47286c32ab"
format_stamp: "Formatted at 2026-09-25 15:11:55 on dist-test-slave-64fx"
server_key: "0e0896988b1578a90e09204a38fd4749"
server_key_iv: "916a0dabf6a56b762ccd423337942728"
server_key_version: "encryptionkey@0"
I20260925 15:11:55.193781 25076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance:
uuid: "2b8d22acf6684427a6c0da47286c32ab"
format_stamp: "Formatted at 2026-09-25 15:11:55 on dist-test-slave-64fx"
server_key: "0e0896988b1578a90e09204a38fd4749"
server_key_iv: "916a0dabf6a56b762ccd423337942728"
server_key_version: "encryptionkey@0"
I20260925 15:11:55.198799 25076 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:11:55.201946 25090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:55.203182 25076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.002s
I20260925 15:11:55.203349 25076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal
uuid: "2b8d22acf6684427a6c0da47286c32ab"
format_stamp: "Formatted at 2026-09-25 15:11:55 on dist-test-slave-64fx"
server_key: "0e0896988b1578a90e09204a38fd4749"
server_key_iv: "916a0dabf6a56b762ccd423337942728"
server_key_version: "encryptionkey@0"
I20260925 15:11:55.203562 25076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:55.226406 25076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:55.227172 25076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:55.227506 25076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:55.228406 25076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:11:55.229840 25076 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:11:55.229959 25076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:55.230098 25076 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:11:55.230186 25076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:55.262183 25076 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.195:37043
I20260925 15:11:55.262315 25202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.195:37043 every 8 connection(s)
I20260925 15:11:55.263410 25076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/data/info.pb
I20260925 15:11:55.271799 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25076
I20260925 15:11:55.271986 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal/instance
I20260925 15:11:55.272558 4919 external_mini_cluster.cc:1428] Setting key 2422bcb2a13f528324230a6012d76d63
I20260925 15:11:55.275206 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.265781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.195:35223 (local address 127.4.205.254:41637)
0925 15:11:55.265947 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.265952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.267217 (+ 1265us) server_negotiation.cc:407] Connection header received
0925 15:11:55.268865 (+ 1648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.268870 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.268939 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.269028 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:55.270060 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.270752 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.272281 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.272895 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.273033 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.274482 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:55.274496 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:55.274503 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:55.274525 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:55.274570 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:55.274586 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:55.274654 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.274884 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.274955 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:11:55.276084 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.266232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41637 (local address 127.4.205.195:35223)
0925 15:11:55.267028 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.267065 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.268611 (+ 1546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.269214 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.269221 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.269302 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:11:55.269888 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.269903 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.270875 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.270879 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.272142 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.272154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.273141 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.273150 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.273401 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.274164 (+ 763us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:11:55.274167 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:11:55.274214 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:11:55.274313 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:11:55.274682 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:11:55.274691 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:11:55.274698 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.274855 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.275187 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":252,"threads_started":1}
I20260925 15:11:55.277218 25203 heartbeater.cc:347] Connected to a master server at 127.4.205.254:41637
I20260925 15:11:55.277489 25203 heartbeater.cc:464] Registering TS with master...
I20260925 15:11:55.278152 25203 heartbeater.cc:511] Master 127.4.205.254:41637 requested a full tablet report, sending...
I20260925 15:11:55.279358 24757 ts_manager.cc:194] Registered new tserver with Master: 2b8d22acf6684427a6c0da47286c32ab (127.4.205.195:37043)
I20260925 15:11:55.280015 24757 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.205.195:35223
I20260925 15:11:55.287719 4919 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:11:55.296804 24795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.291543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.4.205.254:41637)
0925 15:11:55.291722 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.291727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.292143 (+ 416us) server_negotiation.cc:407] Connection header received
0925 15:11:55.292291 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.292294 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.292352 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.292432 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:55.292818 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.293652 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.294919 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.295292 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.295382 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.296250 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:55.296254 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:55.296257 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:55.296273 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:55.296303 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:55.296314 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:55.296380 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.296496 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.296577 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:11:55.300192 24757 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44764:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:11:55.306964 24757 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:11:55.329608 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.318068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:37043 (local address 127.0.0.1:33736)
0925 15:11:55.319571 (+ 1503us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.319624 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.319817 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.322047 (+ 2230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.322058 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.322106 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:55.322450 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.322462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.324609 (+ 2147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.324615 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.327226 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.327240 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.328565 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.328571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.328743 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.328828 (+ 85us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.328965 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.329191 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1197,"mutex_wait_us":181,"thread_start_us":2799,"threads_started":1}
I20260925 15:11:55.329726 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.320622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.193:34509 (local address 127.0.0.1:46382)
0925 15:11:55.321611 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.321628 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.321729 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.322975 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.322979 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.323003 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:55.323331 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.323342 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.325165 (+ 1823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.325170 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.326899 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.326911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.328830 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.328837 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.329112 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.329144 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.329205 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.329327 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":67,"thread_start_us":715,"threads_started":1}
I20260925 15:11:55.330294 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.318375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44833 (local address 127.0.0.1:53924)
0925 15:11:55.321157 (+ 2782us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.321173 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.321318 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.322430 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.322433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.322453 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:55.322742 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.322750 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.324354 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.324360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.327225 (+ 2865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.327240 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.329447 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.329453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.329671 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.329716 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.329810 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.329923 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2660,"mutex_wait_us":105,"thread_start_us":731,"threads_started":1}
I20260925 15:11:55.330559 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.318324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.4.205.195:37043)
0925 15:11:55.320487 (+ 2163us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.320496 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.320531 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:11:55.320619 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.320629 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.320917 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.322659 (+ 1742us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:55.322949 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.324461 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.327481 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.328381 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.328521 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.329894 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.329965 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.330139 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1964,"thread_start_us":1787,"threads_started":1}
I20260925 15:11:55.331034 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.318762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53924 (local address 127.4.205.194:44833)
0925 15:11:55.320725 (+ 1963us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.320735 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.321476 (+ 741us) server_negotiation.cc:407] Connection header received
0925 15:11:55.321629 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.321641 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.321916 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.322140 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:55.323001 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.324219 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.327455 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.328885 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.329055 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.329325 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.330520 (+ 1195us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.330676 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1795,"thread_start_us":1662,"threads_started":1}
I20260925 15:11:55.331277 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.321195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.4.205.193:34509)
0925 15:11:55.322375 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.322384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.322413 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:11:55.322505 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.322513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.322739 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.322968 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:55.323518 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.325012 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.327148 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.328592 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.329846 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.330723 (+ 877us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.330787 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.330937 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":921,"threads_started":1}
I20260925 15:11:55.334261 24878 tablet_service.cc:1467] Processing CreateTablet for tablet 680c87e38d334e6ab4a4a345aae95805 (DEFAULT_TABLE table=test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:55.334463 25008 tablet_service.cc:1467] Processing CreateTablet for tablet 680c87e38d334e6ab4a4a345aae95805 (DEFAULT_TABLE table=test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:55.334923 25138 tablet_service.cc:1467] Processing CreateTablet for tablet 680c87e38d334e6ab4a4a345aae95805 (DEFAULT_TABLE table=test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:11:55.335685 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c87e38d334e6ab4a4a345aae95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:55.335709 24878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c87e38d334e6ab4a4a345aae95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:55.336182 25138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 680c87e38d334e6ab4a4a345aae95805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:55.348294 25221 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Bootstrap starting.
I20260925 15:11:55.350334 25222 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Bootstrap starting.
I20260925 15:11:55.351078 25223 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Bootstrap starting.
I20260925 15:11:55.351976 25221 tablet_bootstrap.cc:654] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:55.353240 25221 log.cc:824] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:55.353950 25222 tablet_bootstrap.cc:654] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:55.354627 25223 tablet_bootstrap.cc:654] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:55.355032 25222 log.cc:824] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:55.355683 25223 log.cc:824] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:55.357278 25221 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: No bootstrap required, opened a new log
I20260925 15:11:55.357649 25221 ts_tablet_manager.cc:1397] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260925 15:11:55.363117 25222 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: No bootstrap required, opened a new log
I20260925 15:11:55.363118 25223 tablet_bootstrap.cc:492] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: No bootstrap required, opened a new log
I20260925 15:11:55.363368 25223 ts_tablet_manager.cc:1397] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260925 15:11:55.363368 25222 ts_tablet_manager.cc:1397] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260925 15:11:55.366724 25221 raft_consensus.cc:348] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.367113 25221 raft_consensus.cc:374] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:55.367233 25221 raft_consensus.cc:729] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32b6fadbb514d21bf233a0a5f718cab, State: Initialized, Role: FOLLOWER
I20260925 15:11:55.367774 25221 consensus_queue.cc:260] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.368759 25073 heartbeater.cc:503] Master 127.4.205.254:41637 was elected leader, sending a full tablet report...
I20260925 15:11:55.369324 25221 ts_tablet_manager.cc:1428] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Time spent starting tablet: real 0.012s user 0.008s sys 0.003s
I20260925 15:11:55.373485 25222 raft_consensus.cc:348] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.373485 25223 raft_consensus.cc:348] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.373937 25223 raft_consensus.cc:374] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:55.373963 25222 raft_consensus.cc:374] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:55.374060 25222 raft_consensus.cc:729] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 769ababfd5944e488c79713cb00a4ccb, State: Initialized, Role: FOLLOWER
I20260925 15:11:55.374060 25223 raft_consensus.cc:729] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8d22acf6684427a6c0da47286c32ab, State: Initialized, Role: FOLLOWER
I20260925 15:11:55.374660 25223 consensus_queue.cc:260] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.374661 25222 consensus_queue.cc:260] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.375732 24943 heartbeater.cc:503] Master 127.4.205.254:41637 was elected leader, sending a full tablet report...
I20260925 15:11:55.376127 25203 heartbeater.cc:503] Master 127.4.205.254:41637 was elected leader, sending a full tablet report...
I20260925 15:11:55.376466 25222 ts_tablet_manager.cc:1428] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 15:11:55.376713 25223 ts_tablet_manager.cc:1428] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Time spent starting tablet: real 0.013s user 0.010s sys 0.004s
W20260925 15:11:55.381137 24944 tablet.cc:2369] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:55.499255 25229 raft_consensus.cc:482] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:11:55.499605 25229 raft_consensus.cc:504] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.501351 25229 leader_election.cc:288] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b8d22acf6684427a6c0da47286c32ab (127.4.205.195:37043), e32b6fadbb514d21bf233a0a5f718cab (127.4.205.194:44833)
I20260925 15:11:55.508662 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.501830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:56823 (local address 127.4.205.195:37043)
0925 15:11:55.502037 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.502043 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.502882 (+ 839us) server_negotiation.cc:407] Connection header received
0925 15:11:55.503063 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.503076 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.503169 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.503258 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:55.503889 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.504627 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.506923 (+ 2296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.507711 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.507866 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.507930 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.508322 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.508435 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:11:55.508698 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.501929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.4.205.193:40905 (local address 127.4.205.194:44833)
0925 15:11:55.502144 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.502152 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.502797 (+ 645us) server_negotiation.cc:407] Connection header received
0925 15:11:55.503000 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.503008 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.503091 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.503171 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:11:55.504001 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.504767 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.506990 (+ 2223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.507787 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.507956 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.508022 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.508323 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.508450 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:11:55.508797 25231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.501920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.195:37043 (local address 127.4.205.193:56823)
0925 15:11:55.502733 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.502772 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.502938 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.503295 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.503301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.503328 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:55.503689 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.503697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.504759 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.504766 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.506702 (+ 1936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.506713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.507888 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.507894 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.508143 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.508167 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.508266 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.508389 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"mutex_wait_us":173,"thread_start_us":507,"threads_started":1}
I20260925 15:11:55.508880 25230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.501874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.194:44833 (local address 127.4.205.193:40905)
0925 15:11:55.502676 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0925 15:11:55.502693 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:11:55.502814 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:11:55.503219 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:11:55.503227 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:11:55.503253 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:11:55.503839 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.503852 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.504904 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.504911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.506702 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:11:55.506713 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.507968 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.507973 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:11:55.508162 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.508190 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 15:11:55.508287 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 15:11:55.508469 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"mutex_wait_us":4,"spinlock_wait_cycles":19840,"thread_start_us":421,"threads_started":1}
I20260925 15:11:55.509876 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c87e38d334e6ab4a4a345aae95805" candidate_uuid: "769ababfd5944e488c79713cb00a4ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8d22acf6684427a6c0da47286c32ab" is_pre_election: true
I20260925 15:11:55.509925 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c87e38d334e6ab4a4a345aae95805" candidate_uuid: "769ababfd5944e488c79713cb00a4ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32b6fadbb514d21bf233a0a5f718cab" is_pre_election: true
I20260925 15:11:55.510411 25028 raft_consensus.cc:2454] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769ababfd5944e488c79713cb00a4ccb in term 0.
I20260925 15:11:55.510442 25158 raft_consensus.cc:2454] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 769ababfd5944e488c79713cb00a4ccb in term 0.
I20260925 15:11:55.511080 24831 leader_election.cc:302] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769ababfd5944e488c79713cb00a4ccb, e32b6fadbb514d21bf233a0a5f718cab; no voters:
I20260925 15:11:55.511418 25229 raft_consensus.cc:2781] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:11:55.511579 25229 raft_consensus.cc:482] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:11:55.511686 25229 raft_consensus.cc:3037] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:55.513839 25229 raft_consensus.cc:504] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.514396 25229 leader_election.cc:288] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [CANDIDATE]: Term 1 election: Requested vote from peers 2b8d22acf6684427a6c0da47286c32ab (127.4.205.195:37043), e32b6fadbb514d21bf233a0a5f718cab (127.4.205.194:44833)
I20260925 15:11:55.514808 25158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c87e38d334e6ab4a4a345aae95805" candidate_uuid: "769ababfd5944e488c79713cb00a4ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b8d22acf6684427a6c0da47286c32ab"
I20260925 15:11:55.514859 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "680c87e38d334e6ab4a4a345aae95805" candidate_uuid: "769ababfd5944e488c79713cb00a4ccb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e32b6fadbb514d21bf233a0a5f718cab"
W20260925 15:11:55.514987 25204 tablet.cc:2369] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:55.515018 25158 raft_consensus.cc:3037] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:55.515046 25028 raft_consensus.cc:3037] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:55.517206 25028 raft_consensus.cc:2454] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769ababfd5944e488c79713cb00a4ccb in term 1.
I20260925 15:11:55.517376 25158 raft_consensus.cc:2454] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 769ababfd5944e488c79713cb00a4ccb in term 1.
I20260925 15:11:55.517619 24831 leader_election.cc:302] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 769ababfd5944e488c79713cb00a4ccb, e32b6fadbb514d21bf233a0a5f718cab; no voters:
I20260925 15:11:55.517910 25229 raft_consensus.cc:2781] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:55.518314 25229 raft_consensus.cc:686] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 1 LEADER]: Becoming Leader. State: Replica: 769ababfd5944e488c79713cb00a4ccb, State: Running, Role: LEADER
I20260925 15:11:55.518810 25229 consensus_queue.cc:237] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } }
I20260925 15:11:55.523147 24756 catalog_manager.cc:5495] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb reported cstate change: term changed from 0 to 1, leader changed from <none> to 769ababfd5944e488c79713cb00a4ccb (127.4.205.193). New cstate: current_term: 1 leader_uuid: "769ababfd5944e488c79713cb00a4ccb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "769ababfd5944e488c79713cb00a4ccb" member_type: VOTER last_known_addr { host: "127.4.205.193" port: 34509 } health_report { overall_health: HEALTHY } } }
I20260925 15:11:55.562285 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.557092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46398 (local address 127.4.205.193:34509)
0925 15:11:55.557279 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.557284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.557410 (+ 126us) server_negotiation.cc:407] Connection header received
0925 15:11:55.557677 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.557681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.557769 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.557913 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:11:55.558268 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.558906 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.560176 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.560553 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.560681 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.561158 (+ 477us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:11:55.561752 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:11:55.561919 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.561965 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.562037 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:11:55.569973 25028 raft_consensus.cc:1261] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 1 FOLLOWER]: Refusing update from remote peer 769ababfd5944e488c79713cb00a4ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:55.569973 25158 raft_consensus.cc:1261] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 1 FOLLOWER]: Refusing update from remote peer 769ababfd5944e488c79713cb00a4ccb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:11:55.571009 25232 consensus_queue.cc:1035] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b8d22acf6684427a6c0da47286c32ab" member_type: VOTER last_known_addr { host: "127.4.205.195" port: 37043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:11:55.571353 25229 consensus_queue.cc:1035] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e32b6fadbb514d21bf233a0a5f718cab" member_type: VOTER last_known_addr { host: "127.4.205.194" port: 44833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:11:55.573333 25074 tablet.cc:2369] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:11:55.581220 25235 mvcc.cc:205] Tried to move back new op lower bound from 7333269977364697088 to 7333269977168945152. Current Snapshot: MvccSnapshot[applied={T|T < 7333269977364697088}]
I20260925 15:11:55.585533 25237 mvcc.cc:205] Tried to move back new op lower bound from 7333269977364697088 to 7333269977168945152. Current Snapshot: MvccSnapshot[applied={T|T < 7333269977364697088}]
I20260925 15:11:55.604096 24757 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:55.604352 24757 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44764:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:11:55.607828 24757 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 861b5c4dff2b47c28eb5a2969396ba4f: Sending DeleteTablet for 3 replicas of tablet 680c87e38d334e6ab4a4a345aae95805
I20260925 15:11:55.609035 25138 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c87e38d334e6ab4a4a345aae95805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:55 UTC) from {username='slave'} at 127.0.0.1:33736
I20260925 15:11:55.609103 24878 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c87e38d334e6ab4a4a345aae95805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:55 UTC) from {username='slave'} at 127.0.0.1:46382
I20260925 15:11:55.609844 25008 tablet_service.cc:1514] Processing DeleteTablet for tablet 680c87e38d334e6ab4a4a345aae95805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:11:55 UTC) from {username='slave'} at 127.0.0.1:53924
I20260925 15:11:55.609985 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24816
I20260925 15:11:55.610227 25247 tablet_replica.cc:330] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: stopping tablet replica
I20260925 15:11:55.610752 25247 raft_consensus.cc:2229] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:55.611364 25247 raft_consensus.cc:2258] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:55.612180 25246 tablet_replica.cc:330] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: stopping tablet replica
I20260925 15:11:55.612654 25246 raft_consensus.cc:2229] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:11:55.613823 25246 raft_consensus.cc:2258] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:55.616256 25246 ts_tablet_manager.cc:1905] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:55.618053 25247 ts_tablet_manager.cc:1905] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:55.622048 25248 tablet_replica.cc:330] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: stopping tablet replica
I20260925 15:11:55.622481 25248 raft_consensus.cc:2229] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:11:55.622969 25248 raft_consensus.cc:2258] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:11:55.624925 25248 ts_tablet_manager.cc:1905] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:11:55.625059 25246 ts_tablet_manager.cc:1918] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:55.625218 25246 log.cc:1196] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-0/wal/wals/680c87e38d334e6ab4a4a345aae95805
I20260925 15:11:55.625825 25246 ts_tablet_manager.cc:1939] T 680c87e38d334e6ab4a4a345aae95805 P 769ababfd5944e488c79713cb00a4ccb: Deleting consensus metadata
I20260925 15:11:55.627480 24742 catalog_manager.cc:4841] TS 769ababfd5944e488c79713cb00a4ccb (127.4.205.193:34509): tablet 680c87e38d334e6ab4a4a345aae95805 (table test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]) successfully deleted
I20260925 15:11:55.630466 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.616126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.4.205.193:34509)
0925 15:11:55.616409 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.616433 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.622242 (+ 5809us) server_negotiation.cc:407] Connection header received
0925 15:11:55.622451 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.622456 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.622552 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.622660 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:55.623354 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.624614 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.626824 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.627476 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.627587 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.628921 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:55.628937 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:55.628950 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:55.629497 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:55.629541 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:55.629681 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:55.629782 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.629988 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.630085 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168}
I20260925 15:11:55.631721 24938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:55.632172 25248 ts_tablet_manager.cc:1918] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:55.632383 25248 log.cc:1196] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-1/wal/wals/680c87e38d334e6ab4a4a345aae95805
I20260925 15:11:55.632463 25247 ts_tablet_manager.cc:1918] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:11:55.632643 25247 log.cc:1196] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Deleting WAL directory at /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348808911781-4919-0/minicluster-data/ts-2/wal/wals/680c87e38d334e6ab4a4a345aae95805
I20260925 15:11:55.633054 25248 ts_tablet_manager.cc:1939] T 680c87e38d334e6ab4a4a345aae95805 P e32b6fadbb514d21bf233a0a5f718cab: Deleting consensus metadata
I20260925 15:11:55.633314 25247 ts_tablet_manager.cc:1939] T 680c87e38d334e6ab4a4a345aae95805 P 2b8d22acf6684427a6c0da47286c32ab: Deleting consensus metadata
I20260925 15:11:55.634766 24743 catalog_manager.cc:4841] TS 2b8d22acf6684427a6c0da47286c32ab (127.4.205.195:37043): tablet 680c87e38d334e6ab4a4a345aae95805 (table test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]) successfully deleted
I20260925 15:11:55.634997 24741 catalog_manager.cc:4841] TS e32b6fadbb514d21bf233a0a5f718cab (127.4.205.194:44833): tablet 680c87e38d334e6ab4a4a345aae95805 (table test-table [id=e9f32104cb8c4b96ad9a278dc4118d72]) successfully deleted
I20260925 15:11:55.725889 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24816
I20260925 15:11:55.742561 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24946
I20260925 15:11:55.748672 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.743289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53940 (local address 127.4.205.194:44833)
0925 15:11:55.743435 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.743439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.743458 (+ 19us) server_negotiation.cc:407] Connection header received
0925 15:11:55.743598 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.743601 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.743737 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.743821 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:55.744171 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.744836 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.746147 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.746532 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.746620 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.747523 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:55.747531 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:55.747544 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:55.748094 (+ 550us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:55.748139 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:55.748162 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:55.748250 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.748352 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.748431 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:11:55.749274 25068 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:55.835119 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24946
I20260925 15:11:55.852507 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25076
I20260925 15:11:55.859140 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:55.853223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33748 (local address 127.4.205.195:37043)
0925 15:11:55.853402 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:55.853409 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:55.853434 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:11:55.853706 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:55.853711 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:55.853790 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:55.853907 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:55.854313 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.855054 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.856308 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:55.856714 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:55.856841 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:55.857881 (+ 1040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:55.857889 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:55.857904 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:55.858446 (+ 542us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:55.858489 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:55.858507 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:55.858635 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:55.858753 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:11:55.858860 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:11:55.859752 25198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:55.957067 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25076
I20260925 15:11:55.972477 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 24726
I20260925 15:11:55.973524 24787 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:56.079366 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 24726
2026-09-25T15:11:56Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2662 ms)
[----------] 6 tests from EncryptionPolicyTest (20868 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:56 dist-test-slave-64fx krb5kdc[25256](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:56 dist-test-slave-64fx krb5kdc[25256](info): set up 2 sockets
Sep 25 15:11:56 dist-test-slave-64fx krb5kdc[25256](info): commencing operation
krb5kdc: starting...
W20260925 15:11:57.150040 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:11:57 dist-test-slave-64fx krb5kdc[25256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:11:57.362905 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43087
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:57.667428 25269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:57.667692 25269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:57.667855 25269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:57.674667 25269 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:57.674788 25269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:57.674870 25269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:57.674966 25269 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:57.675189 25269 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:57.684674 25269 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43087
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43087
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:57.685931 25269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:57.687144 25269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:57.698982 25275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:57.699046 25277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:57.699070 25274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:57.700155 25269 server_base.cc:1034] running on GCE node
I20260925 15:11:57.700919 25269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:57.702306 25269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:57.703509 25269 hybrid_clock.cc:648] HybridClock initialized: now 1790349117703466 us; error 33 us; skew 500 ppm
I20260925 15:11:57.703805 25269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:57 dist-test-slave-64fx krb5kdc[25256](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349117, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:57.706734 25269 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:11:57.708175 25269 webserver.cc:460] Webserver started at http://127.0.0.1:44741/ using document root <none> and password file <none>
I20260925 15:11:57.708825 25269 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:57.708949 25269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:57.709255 25269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:57.712267 25269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "b569123902e945c6ba3df6ecc8ab9960"
format_stamp: "Formatted at 2026-09-25 15:11:57 on dist-test-slave-64fx"
server_key: "bcbe10f081c99e3b9ee7becdef2617aa"
server_key_iv: "5a2f01b262d5e37e3fb2d2f8041c4443"
server_key_version: "encryptionkey@0"
I20260925 15:11:57.713025 25269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "b569123902e945c6ba3df6ecc8ab9960"
format_stamp: "Formatted at 2026-09-25 15:11:57 on dist-test-slave-64fx"
server_key: "bcbe10f081c99e3b9ee7becdef2617aa"
server_key_iv: "5a2f01b262d5e37e3fb2d2f8041c4443"
server_key_version: "encryptionkey@0"
I20260925 15:11:57.717811 25269 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.001s
I20260925 15:11:57.720741 25284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:57.722023 25269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:11:57.722195 25269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "b569123902e945c6ba3df6ecc8ab9960"
format_stamp: "Formatted at 2026-09-25 15:11:57 on dist-test-slave-64fx"
server_key: "bcbe10f081c99e3b9ee7becdef2617aa"
server_key_iv: "5a2f01b262d5e37e3fb2d2f8041c4443"
server_key_version: "encryptionkey@0"
I20260925 15:11:57.722376 25269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:57.737727 25269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:57.740958 25269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:57.741261 25269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:57.758134 25269 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43087
I20260925 15:11:57.758168 25335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43087 every 8 connection(s)
I20260925 15:11:57.759297 25269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:57.762603 25336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:57.763172 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25269
I20260925 15:11:57.763365 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:57.764106 4919 external_mini_cluster.cc:1428] Setting key 96943adaabe3b411b4cd94e7c50c3d80
I20260925 15:11:57.773902 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960: Bootstrap starting.
Sep 25 15:11:57 dist-test-slave-64fx krb5kdc[25256](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:11:57.776491 25336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:57.777415 25336 log.cc:824] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:57.779860 25336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960: No bootstrap required, opened a new log
I20260925 15:11:57.784451 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:57.765841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55963 (local address 127.0.0.1:43087)
0925 15:11:57.766541 (+ 700us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:57.766550 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:57.766585 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:11:57.768294 (+ 1709us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:57.768313 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:57.768639 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:57.769028 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:57.770436 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:57.771569 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:57.772843 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:57.773459 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:57.773672 (+ 213us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:57.777425 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:57.777434 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:57.777446 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:57.777475 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:57.780279 (+ 2804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:57.781063 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:57.781069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:57.781074 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:57.781234 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:57.781801 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:57.781805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:57.781807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:57.782438 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:57.782780 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:57.783300 (+ 520us) server_negotiation.cc:299] Negotiation successful
0925 15:11:57.783566 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":527,"thread_start_us":319,"threads_started":1}
I20260925 15:11:57.788136 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:11:57.788067 25336 raft_consensus.cc:348] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:11:57.788442 25336 raft_consensus.cc:374] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:57.788534 25336 raft_consensus.cc:729] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b569123902e945c6ba3df6ecc8ab9960, State: Initialized, Role: FOLLOWER
I20260925 15:11:57.789098 25336 consensus_queue.cc:260] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:11:57.789331 25336 raft_consensus.cc:388] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:57.789470 25336 raft_consensus.cc:482] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:57.789681 25336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:57.791836 25336 raft_consensus.cc:504] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:11:57.792286 25336 leader_election.cc:302] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b569123902e945c6ba3df6ecc8ab9960; no voters:
I20260925 15:11:57.792855 25336 leader_election.cc:288] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:57.792996 25345 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:57.794696 25336 sys_catalog.cc:564] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:57.798542 25345 raft_consensus.cc:686] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [term 1 LEADER]: Becoming Leader. State: Replica: b569123902e945c6ba3df6ecc8ab9960, State: Running, Role: LEADER
I20260925 15:11:57.799093 25345 consensus_queue.cc:237] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } }
I20260925 15:11:57.802021 25348 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b569123902e945c6ba3df6ecc8ab9960. Latest consensus state: current_term: 1 leader_uuid: "b569123902e945c6ba3df6ecc8ab9960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260925 15:11:57.802356 25348 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:57.802855 25347 sys_catalog.cc:455] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b569123902e945c6ba3df6ecc8ab9960" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b569123902e945c6ba3df6ecc8ab9960" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43087 } } }
I20260925 15:11:57.803124 25347 sys_catalog.cc:458] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960 [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:57.807338 25356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:57.812388 25356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:57.815290 25339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:57.793024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42733 (local address 127.0.0.1:43087)
0925 15:11:57.793175 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:57.793182 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:57.798300 (+ 5118us) server_negotiation.cc:407] Connection header received
0925 15:11:57.799130 (+ 830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:57.799135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:57.799210 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:57.799328 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:57.801348 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:57.802642 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:57.804547 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:57.805511 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:57.805655 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:57.808644 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:57.808648 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:57.808654 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:11:57.808692 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:57.812049 (+ 3357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:57.812909 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:57.812915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:57.812921 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:57.813088 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:11:57.813608 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:11:57.813612 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:11:57.813615 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:11:57.814088 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:57.814353 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:57.814777 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 15:11:57.814947 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:11:57.820399 25356 catalog_manager.cc:1347] Generated new cluster ID: 07e521e019bb41938c37e4d08fe7b129
I20260925 15:11:57.820510 25356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:57.830354 25356 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:57.831364 25356 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:57.837313 25356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b569123902e945c6ba3df6ecc8ab9960: Generated new TSK 0
I20260925 15:11:57.838027 25356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:11:57.847259 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25269
I20260925 15:11:57.848294 25331 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:11:57.933495 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25269
W20260925 15:11:57.949522 25254 connection.cc:535] client connection to 127.0.0.1:43087 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43087): Connection reset by peer (error 104)
2026-09-25T15:11:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1853 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:58 dist-test-slave-64fx krb5kdc[25369](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:58 dist-test-slave-64fx krb5kdc[25369](info): set up 2 sockets
Sep 25 15:11:58 dist-test-slave-64fx krb5kdc[25369](info): commencing operation
krb5kdc: starting...
W20260925 15:11:58.998101 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:11:59 dist-test-slave-64fx krb5kdc[25369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:11:59.236433 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43959
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:11:59.557542 25382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:11:59.557864 25382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:11:59.558027 25382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:11:59.564520 25382 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:11:59.564661 25382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:11:59.564770 25382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:11:59.564874 25382 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:11:59.564970 25382 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:11:59.573868 25382 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:11:59.575013 25382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:11:59.576231 25382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:11:59.587112 25387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:59.587080 25390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:11:59.587141 25388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:11:59.588136 25382 server_base.cc:1034] running on GCE node
I20260925 15:11:59.588778 25382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:11:59.589988 25382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:11:59.591197 25382 hybrid_clock.cc:648] HybridClock initialized: now 1790349119591149 us; error 40 us; skew 500 ppm
I20260925 15:11:59.591499 25382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:11:59 dist-test-slave-64fx krb5kdc[25369](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:11:59.594307 25382 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:11:59.595520 25382 webserver.cc:460] Webserver started at http://127.0.0.1:37437/ using document root <none> and password file <none>
I20260925 15:11:59.596174 25382 fs_manager.cc:359] Metadata directory not provided
I20260925 15:11:59.596304 25382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:11:59.596617 25382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:11:59.599315 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee"
format_stamp: "Formatted at 2026-09-25 15:11:59 on dist-test-slave-64fx"
server_key: "0df960d72b11c5f76c35476185e5915f"
server_key_iv: "368aa566bb3cd84ae2f08bc2d3ac315c"
server_key_version: "encryptionkey@0"
I20260925 15:11:59.600010 25382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee"
format_stamp: "Formatted at 2026-09-25 15:11:59 on dist-test-slave-64fx"
server_key: "0df960d72b11c5f76c35476185e5915f"
server_key_iv: "368aa566bb3cd84ae2f08bc2d3ac315c"
server_key_version: "encryptionkey@0"
I20260925 15:11:59.604569 25382 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260925 15:11:59.607652 25397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:11:59.608708 25382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:11:59.608915 25382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee"
format_stamp: "Formatted at 2026-09-25 15:11:59 on dist-test-slave-64fx"
server_key: "0df960d72b11c5f76c35476185e5915f"
server_key_iv: "368aa566bb3cd84ae2f08bc2d3ac315c"
server_key_version: "encryptionkey@0"
I20260925 15:11:59.609097 25382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:11:59.632797 25382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:11:59.636222 25382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:11:59.636487 25382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:11:59.653548 25382 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43959
I20260925 15:11:59.653604 25448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43959 every 8 connection(s)
I20260925 15:11:59.654773 25382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:11:59.656863 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25382
I20260925 15:11:59.657049 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:11:59.657496 4919 external_mini_cluster.cc:1428] Setting key 27d34afd013befdd461f6d4bafcfbb75
I20260925 15:11:59.658388 25449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:11:59.671108 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:59.659188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34811 (local address 127.0.0.1:43959)
0925 15:11:59.659924 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:59.659937 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:59.659979 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:11:59.661744 (+ 1765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:59.661770 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:59.662269 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:59.662668 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:59.663154 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:59.665089 (+ 1935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:59.666353 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:59.667308 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:59.667477 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:59.668550 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:59.668561 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:59.668583 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:59.669242 (+ 659us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:59.669324 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:59.669355 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:59.669509 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:59.669734 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:11:59.669931 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":296,"threads_started":1}
I20260925 15:11:59.672400 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee: Bootstrap starting.
I20260925 15:11:59.674557 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:11:59.675568 25449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee: Neither blocks nor log segments found. Creating new log.
I20260925 15:11:59.676683 25449 log.cc:824] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee: Log is configured to *not* fsync() on all Append() calls
I20260925 15:11:59.679349 25449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee: No bootstrap required, opened a new log
I20260925 15:11:59.686293 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:11:59.679778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45927 (local address 127.0.0.1:43959)
0925 15:11:59.679974 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:11:59.679980 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:11:59.680201 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:11:59.680354 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:11:59.680358 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:11:59.680410 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:11:59.680485 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:11:59.681118 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:59.681946 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:59.683821 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:11:59.684241 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:11:59.684323 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:11:59.685434 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:11:59.685439 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:11:59.685442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:11:59.685460 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:11:59.685505 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:11:59.685518 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:11:59.685712 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 15:11:59.685904 (+ 192us) server_negotiation.cc:299] Negotiation successful
0925 15:11:59.685969 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:11:59.689270 25449 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } }
I20260925 15:11:59.689688 25449 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:11:59.689800 25449 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7bce2d79bc149778e4c4e1aa8ef60ee, State: Initialized, Role: FOLLOWER
I20260925 15:11:59.690436 25449 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } }
I20260925 15:11:59.690722 25449 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:11:59.690867 25449 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:11:59.691051 25449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:11:59.693089 25449 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } }
I20260925 15:11:59.693589 25449 leader_election.cc:302] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7bce2d79bc149778e4c4e1aa8ef60ee; no voters:
I20260925 15:11:59.694144 25449 leader_election.cc:288] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:11:59.694278 25459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:11:59.694983 25459 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [term 1 LEADER]: Becoming Leader. State: Replica: f7bce2d79bc149778e4c4e1aa8ef60ee, State: Running, Role: LEADER
I20260925 15:11:59.695506 25459 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } }
I20260925 15:11:59.696110 25449 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:11:59.698045 25460 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } } }
I20260925 15:11:59.698406 25460 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:59.698918 25461 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7bce2d79bc149778e4c4e1aa8ef60ee. Latest consensus state: current_term: 1 leader_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7bce2d79bc149778e4c4e1aa8ef60ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43959 } } }
I20260925 15:11:59.699208 25461 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee [sys.catalog]: This master's current role is: LEADER
I20260925 15:11:59.699414 25465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:11:59.704770 25465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:11:59.712886 25465 catalog_manager.cc:1347] Generated new cluster ID: 4b089971d8bb473da61665bbe9c2f804
I20260925 15:11:59.713033 25465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:11:59.734503 25465 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:11:59.735579 25465 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:11:59.753906 25465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7bce2d79bc149778e4c4e1aa8ef60ee: Generated new TSK 0
I20260925 15:11:59.754663 25465 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:11:59.770583 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25382
W20260925 15:11:59.771603 25444 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34811
W20260925 15:11:59.772344 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25382: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:11:59.772532 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25382
W20260925 15:11:59.789209 25367 connection.cc:535] client connection to 127.0.0.1:43959 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43959): Connection reset by peer (error 104)
2026-09-25T15:11:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1835 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:11:59 dist-test-slave-64fx krb5kdc[25480](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:11:59 dist-test-slave-64fx krb5kdc[25480](info): set up 2 sockets
Sep 25 15:11:59 dist-test-slave-64fx krb5kdc[25480](info): commencing operation
krb5kdc: starting...
W20260925 15:12:00.817761 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:00 dist-test-slave-64fx krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349120, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:01.042302 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:01.351218 25493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:01.351534 25493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:01.351704 25493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:01.359743 25493 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:01.359872 25493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:01.359951 25493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:01.360029 25493 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:01.360088 25493 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:01.369165 25493 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43335
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:01.370388 25493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:01.371629 25493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:01.382761 25499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:01.382812 25501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:01.382827 25498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:01.383467 25493 server_base.cc:1034] running on GCE node
I20260925 15:12:01.384152 25493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:01.385357 25493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:01.386574 25493 hybrid_clock.cc:648] HybridClock initialized: now 1790349121386507 us; error 74 us; skew 500 ppm
I20260925 15:12:01.386909 25493 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:01 dist-test-slave-64fx krb5kdc[25480](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:01.392820 25493 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:01.394246 25493 webserver.cc:460] Webserver started at http://127.0.0.1:39929/ using document root <none> and password file <none>
I20260925 15:12:01.394884 25493 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:01.395018 25493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:01.395351 25493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:01.398080 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "fc25d4b5b7a9422cae725db468215aed"
format_stamp: "Formatted at 2026-09-25 15:12:01 on dist-test-slave-64fx"
server_key: "61e2c50e2d3e504f457c2aab63413700"
server_key_iv: "ebe344cf5d964b350659311b3c40d443"
server_key_version: "encryptionkey@0"
I20260925 15:12:01.398826 25493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "fc25d4b5b7a9422cae725db468215aed"
format_stamp: "Formatted at 2026-09-25 15:12:01 on dist-test-slave-64fx"
server_key: "61e2c50e2d3e504f457c2aab63413700"
server_key_iv: "ebe344cf5d964b350659311b3c40d443"
server_key_version: "encryptionkey@0"
I20260925 15:12:01.403539 25493 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:01.406737 25508 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:01.407903 25493 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:12:01.408073 25493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "fc25d4b5b7a9422cae725db468215aed"
format_stamp: "Formatted at 2026-09-25 15:12:01 on dist-test-slave-64fx"
server_key: "61e2c50e2d3e504f457c2aab63413700"
server_key_iv: "ebe344cf5d964b350659311b3c40d443"
server_key_version: "encryptionkey@0"
I20260925 15:12:01.408255 25493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:01.413508 25493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:01.417232 25493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:01.417508 25493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:01.435320 25493 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43335
I20260925 15:12:01.435369 25559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43335 every 8 connection(s)
I20260925 15:12:01.436687 25493 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:01.440310 25560 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:01.441689 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25493
I20260925 15:12:01.441865 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:01.442267 4919 external_mini_cluster.cc:1428] Setting key 4bc8ef2407147a656f560081496b1d2a
I20260925 15:12:01.449720 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:01.443918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46163 (local address 127.0.0.1:43335)
0925 15:12:01.444730 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:01.444741 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:01.444781 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:12:01.446439 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:01.446461 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:01.446805 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:01.447141 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:01.448179 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:01.448185 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:01.448198 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:01.448224 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:01.448285 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:01.448303 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:01.448451 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:01.448591 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:12:01.448795 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":350,"threads_started":1}
I20260925 15:12:01.453352 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:01.453892 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed: Bootstrap starting.
I20260925 15:12:01.456790 25560 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:01.458082 25560 log.cc:824] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:01.460091 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:01.457700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37323 (local address 127.0.0.1:43335)
0925 15:12:01.457874 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:01.457884 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:01.458106 (+ 222us) server_negotiation.cc:407] Connection header received
0925 15:12:01.458324 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:01.458329 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:01.458383 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:01.458504 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:01.459440 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:01.459446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:01.459450 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:01.459467 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:01.459509 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:01.459534 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:01.459647 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:01.459748 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 15:12:01.459811 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 15:12:01.461107 25560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed: No bootstrap required, opened a new log
I20260925 15:12:01.468495 25560 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } }
I20260925 15:12:01.468832 25560 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:01.468920 25560 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc25d4b5b7a9422cae725db468215aed, State: Initialized, Role: FOLLOWER
I20260925 15:12:01.469516 25560 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } }
I20260925 15:12:01.469800 25560 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:01.469949 25560 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:01.470112 25560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:01.472500 25560 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } }
I20260925 15:12:01.472997 25560 leader_election.cc:302] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc25d4b5b7a9422cae725db468215aed; no voters:
I20260925 15:12:01.473486 25560 leader_election.cc:288] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:01.473636 25570 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:01.474233 25570 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [term 1 LEADER]: Becoming Leader. State: Replica: fc25d4b5b7a9422cae725db468215aed, State: Running, Role: LEADER
I20260925 15:12:01.474699 25570 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } }
I20260925 15:12:01.475374 25560 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:01.477356 25572 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc25d4b5b7a9422cae725db468215aed. Latest consensus state: current_term: 1 leader_uuid: "fc25d4b5b7a9422cae725db468215aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } } }
I20260925 15:12:01.477790 25572 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:01.478442 25571 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc25d4b5b7a9422cae725db468215aed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc25d4b5b7a9422cae725db468215aed" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43335 } } }
I20260925 15:12:01.478957 25571 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:01.480242 25577 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:01.485410 25577 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:01.493068 25577 catalog_manager.cc:1347] Generated new cluster ID: 7ac86dc2e8d44866ad06c0b122b24923
I20260925 15:12:01.493204 25577 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:01.517354 25577 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:01.517840 25577 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:01.524910 25577 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc25d4b5b7a9422cae725db468215aed: Generated new TSK 0
I20260925 15:12:01.525733 25577 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:01.531122 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25493
W20260925 15:12:01.532258 25555 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46163
W20260925 15:12:01.532907 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25493: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:01.533059 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25493
2026-09-25T15:12:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1759 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:01 dist-test-slave-64fx krb5kdc[25591](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:01 dist-test-slave-64fx krb5kdc[25591](info): set up 2 sockets
Sep 25 15:12:01 dist-test-slave-64fx krb5kdc[25591](info): commencing operation
krb5kdc: starting...
W20260925 15:12:02.601651 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.002s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:02 dist-test-slave-64fx krb5kdc[25591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:02.820494 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:03.130641 25604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:03.130918 25604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:03.131034 25604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:03.137786 25604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:03.137880 25604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:03.137924 25604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:03.137990 25604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:03.138032 25604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:03.146965 25604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:03.148125 25604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:03.149307 25604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:03.160516 25610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:03.160602 25609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:03.160571 25612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:03.161399 25604 server_base.cc:1034] running on GCE node
I20260925 15:12:03.162137 25604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:03.163384 25604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:03.164572 25604 hybrid_clock.cc:648] HybridClock initialized: now 1790349123164518 us; error 53 us; skew 500 ppm
I20260925 15:12:03.164901 25604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:03 dist-test-slave-64fx krb5kdc[25591](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:03.167910 25604 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:03.169276 25604 webserver.cc:460] Webserver started at http://127.0.0.1:43115/ using document root <none> and password file <none>
I20260925 15:12:03.170007 25604 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:03.170143 25604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:03.170432 25604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:03.173171 25604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "c8e0d85f6fa946708025a340e0bec27a"
format_stamp: "Formatted at 2026-09-25 15:12:03 on dist-test-slave-64fx"
server_key: "29d7783faf298d20ef134fed8aa5362e"
server_key_iv: "de2f134bbcecae671a1c3c8b70141321"
server_key_version: "encryptionkey@0"
I20260925 15:12:03.173978 25604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "c8e0d85f6fa946708025a340e0bec27a"
format_stamp: "Formatted at 2026-09-25 15:12:03 on dist-test-slave-64fx"
server_key: "29d7783faf298d20ef134fed8aa5362e"
server_key_iv: "de2f134bbcecae671a1c3c8b70141321"
server_key_version: "encryptionkey@0"
I20260925 15:12:03.178857 25604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:12:03.182140 25619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:03.183384 25604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 15:12:03.183585 25604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "c8e0d85f6fa946708025a340e0bec27a"
format_stamp: "Formatted at 2026-09-25 15:12:03 on dist-test-slave-64fx"
server_key: "29d7783faf298d20ef134fed8aa5362e"
server_key_iv: "de2f134bbcecae671a1c3c8b70141321"
server_key_version: "encryptionkey@0"
I20260925 15:12:03.183794 25604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:03.198570 25604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:03.202663 25604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:03.202937 25604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:03.221668 25604 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39875
I20260925 15:12:03.221704 25670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39875 every 8 connection(s)
I20260925 15:12:03.223037 25604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:03.226586 25671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:03.230484 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25604
I20260925 15:12:03.230664 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:03.231190 4919 external_mini_cluster.cc:1428] Setting key 03fd52158503a70ac53965c7a08f1c04
I20260925 15:12:03.239267 25671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a: Bootstrap starting.
I20260925 15:12:03.243047 25671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:03.244422 25671 log.cc:824] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:12:03 dist-test-slave-64fx krb5kdc[25591](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:12:03.247812 25671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a: No bootstrap required, opened a new log
I20260925 15:12:03.253756 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:03.232876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43551 (local address 127.0.0.1:39875)
0925 15:12:03.233684 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:03.233696 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:03.233735 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:12:03.235314 (+ 1579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:03.235335 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:03.235707 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:03.236004 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:03.237731 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:03.238963 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:03.240688 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:03.241380 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:03.241510 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:03.246700 (+ 5190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:03.246707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:03.246720 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:03.246752 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:03.249779 (+ 3027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:03.250632 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:03.250638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:03.250644 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:03.250803 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:03.251179 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:03.251182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:03.251184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:03.251680 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:03.251904 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:03.252326 (+ 422us) server_negotiation.cc:299] Negotiation successful
0925 15:12:03.252668 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":368,"threads_started":1}
I20260925 15:12:03.257524 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:03.257285 25671 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } }
I20260925 15:12:03.257860 25671 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:03.257958 25671 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8e0d85f6fa946708025a340e0bec27a, State: Initialized, Role: FOLLOWER
I20260925 15:12:03.258582 25671 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } }
I20260925 15:12:03.258818 25671 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:03.258953 25671 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:03.259109 25671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:03.261377 25671 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } }
I20260925 15:12:03.261905 25671 leader_election.cc:302] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8e0d85f6fa946708025a340e0bec27a; no voters:
I20260925 15:12:03.263126 25671 leader_election.cc:288] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:03.263228 25681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:03.265036 25671 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:03.265815 25681 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [term 1 LEADER]: Becoming Leader. State: Replica: c8e0d85f6fa946708025a340e0bec27a, State: Running, Role: LEADER
I20260925 15:12:03.266471 25681 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } }
I20260925 15:12:03.269701 25683 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8e0d85f6fa946708025a340e0bec27a. Latest consensus state: current_term: 1 leader_uuid: "c8e0d85f6fa946708025a340e0bec27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } } }
I20260925 15:12:03.270143 25683 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:03.269649 25682 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8e0d85f6fa946708025a340e0bec27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8e0d85f6fa946708025a340e0bec27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39875 } } }
I20260925 15:12:03.270269 25682 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:03.273845 25688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:03.280804 25688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:03.281184 25674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:03.262203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58137 (local address 127.0.0.1:39875)
0925 15:12:03.262402 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:03.262411 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:03.262785 (+ 374us) server_negotiation.cc:407] Connection header received
0925 15:12:03.262928 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:03.262936 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:03.263029 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:03.263145 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:03.265292 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:03.266635 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:03.268489 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:03.269387 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:03.269548 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:03.272830 (+ 3282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:03.272838 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:03.272842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:03.272887 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:03.277231 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:03.278126 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:03.278133 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:03.278140 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:03.278284 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:03.278756 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:03.278762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:03.278765 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:03.279177 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:03.279408 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:03.280483 (+ 1075us) server_negotiation.cc:299] Negotiation successful
0925 15:12:03.280718 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:12:03.291838 25688 catalog_manager.cc:1347] Generated new cluster ID: 49dc7fc9a75d473699cafeb134c27842
I20260925 15:12:03.291941 25688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:03.300149 25688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:03.301993 25688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:03.309175 25688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8e0d85f6fa946708025a340e0bec27a: Generated new TSK 0
I20260925 15:12:03.309963 25688 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:03.319702 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25604
I20260925 15:12:03.320595 25666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:03.407936 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25604
2026-09-25T15:12:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1876 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:03 dist-test-slave-64fx krb5kdc[25703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:03 dist-test-slave-64fx krb5kdc[25703](info): set up 2 sockets
Sep 25 15:12:03 dist-test-slave-64fx krb5kdc[25703](info): commencing operation
krb5kdc: starting...
W20260925 15:12:04.469635 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:04 dist-test-slave-64fx krb5kdc[25703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:04.692301 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:05.006373 25716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:05.006664 25716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:05.006755 25716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:05.013886 25716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:05.013993 25716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:05.014036 25716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:05.014097 25716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:05.014140 25716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:05.023455 25716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:05.024655 25716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:05.025964 25716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:05.039613 25722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:05.039862 25721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:05.039901 25716 server_base.cc:1034] running on GCE node
W20260925 15:12:05.040417 25724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:05.041450 25716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:05.042851 25716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:05.044044 25716 hybrid_clock.cc:648] HybridClock initialized: now 1790349125043988 us; error 55 us; skew 500 ppm
I20260925 15:12:05.044374 25716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:05 dist-test-slave-64fx krb5kdc[25703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349125, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:05.047390 25716 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:05.048729 25716 webserver.cc:460] Webserver started at http://127.0.0.1:33515/ using document root <none> and password file <none>
I20260925 15:12:05.049352 25716 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:05.049441 25716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:05.049737 25716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:05.052590 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "a559af2b898f40d8854285e2e8595ae0"
format_stamp: "Formatted at 2026-09-25 15:12:05 on dist-test-slave-64fx"
server_key: "a9f79d022910806b1377337fc3739142"
server_key_iv: "ad2edaeb061209932a317204e4e77bdc"
server_key_version: "encryptionkey@0"
I20260925 15:12:05.053264 25716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "a559af2b898f40d8854285e2e8595ae0"
format_stamp: "Formatted at 2026-09-25 15:12:05 on dist-test-slave-64fx"
server_key: "a9f79d022910806b1377337fc3739142"
server_key_iv: "ad2edaeb061209932a317204e4e77bdc"
server_key_version: "encryptionkey@0"
I20260925 15:12:05.058002 25716 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 15:12:05.061020 25731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:05.062354 25716 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:05.062528 25716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "a559af2b898f40d8854285e2e8595ae0"
format_stamp: "Formatted at 2026-09-25 15:12:05 on dist-test-slave-64fx"
server_key: "a9f79d022910806b1377337fc3739142"
server_key_iv: "ad2edaeb061209932a317204e4e77bdc"
server_key_version: "encryptionkey@0"
I20260925 15:12:05.062709 25716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:05.083287 25716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:05.086831 25716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:05.087111 25716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:05.104132 25716 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44385
I20260925 15:12:05.104171 25782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44385 every 8 connection(s)
I20260925 15:12:05.105381 25716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:05.108898 25783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:05.112412 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25716
I20260925 15:12:05.112598 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:05.113054 4919 external_mini_cluster.cc:1428] Setting key 83ddb728033aaa41395d1955e959bb68
I20260925 15:12:05.121163 25783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0: Bootstrap starting.
I20260925 15:12:05.124676 25783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:05.125810 25783 log.cc:824] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:05.126000 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:05.114930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37301 (local address 127.0.0.1:44385)
0925 15:12:05.115666 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:05.115676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:05.115715 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:12:05.117280 (+ 1565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:05.117300 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:05.117754 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:05.118064 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:05.118659 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:05.119925 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:05.121843 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:05.122460 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:05.122611 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:05.123745 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:05.123751 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:05.123763 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:05.124274 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:05.124349 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:05.124371 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:05.124473 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:05.124709 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 15:12:05.124972 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":328,"threads_started":1}
I20260925 15:12:05.128674 25783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0: No bootstrap required, opened a new log
I20260925 15:12:05.129379 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:05.137187 25783 raft_consensus.cc:348] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } }
I20260925 15:12:05.137622 25783 raft_consensus.cc:374] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:05.137756 25783 raft_consensus.cc:729] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a559af2b898f40d8854285e2e8595ae0, State: Initialized, Role: FOLLOWER
I20260925 15:12:05.138408 25783 consensus_queue.cc:260] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } }
I20260925 15:12:05.138710 25783 raft_consensus.cc:388] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:05.138859 25783 raft_consensus.cc:482] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:05.139075 25783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:05.139662 25786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:05.133292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49741 (local address 127.0.0.1:44385)
0925 15:12:05.133480 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:05.133489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:05.133963 (+ 474us) server_negotiation.cc:407] Connection header received
0925 15:12:05.134145 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:05.134148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:05.134199 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:05.134272 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:05.134809 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:05.135493 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:05.137306 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:05.137773 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:05.137885 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:05.138927 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:05.138931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:05.138934 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:05.138949 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:05.138981 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:05.139016 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:05.139111 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:05.139292 (+ 181us) server_negotiation.cc:299] Negotiation successful
0925 15:12:05.139359 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:12:05.141522 25783 raft_consensus.cc:504] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } }
I20260925 15:12:05.142047 25783 leader_election.cc:302] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a559af2b898f40d8854285e2e8595ae0; no voters:
I20260925 15:12:05.142596 25783 leader_election.cc:288] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:05.142696 25793 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:05.144259 25793 raft_consensus.cc:686] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [term 1 LEADER]: Becoming Leader. State: Replica: a559af2b898f40d8854285e2e8595ae0, State: Running, Role: LEADER
I20260925 15:12:05.144605 25783 sys_catalog.cc:564] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:05.144774 25793 consensus_queue.cc:237] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } }
I20260925 15:12:05.147397 25795 sys_catalog.cc:455] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a559af2b898f40d8854285e2e8595ae0. Latest consensus state: current_term: 1 leader_uuid: "a559af2b898f40d8854285e2e8595ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } } }
I20260925 15:12:05.147413 25794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a559af2b898f40d8854285e2e8595ae0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a559af2b898f40d8854285e2e8595ae0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44385 } } }
I20260925 15:12:05.147795 25794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:05.147795 25795 sys_catalog.cc:458] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:05.149240 25800 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:05.154234 25800 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:05.162657 25800 catalog_manager.cc:1347] Generated new cluster ID: 82e996a053104b76a914c92d913b4abf
I20260925 15:12:05.162761 25800 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:05.177786 25800 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:05.178995 25800 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:05.193379 25800 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a559af2b898f40d8854285e2e8595ae0: Generated new TSK 0
I20260925 15:12:05.194224 25800 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:05.207676 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25716
W20260925 15:12:05.208961 25778 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37301
W20260925 15:12:05.209640 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25716: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:05.209782 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25716
2026-09-25T15:12:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1802 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:05 dist-test-slave-64fx krb5kdc[25814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:05 dist-test-slave-64fx krb5kdc[25814](info): set up 2 sockets
Sep 25 15:12:05 dist-test-slave-64fx krb5kdc[25814](info): commencing operation
krb5kdc: starting...
W20260925 15:12:06.273849 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:06 dist-test-slave-64fx krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:06.504057 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37875
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:06.813181 25827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:06.813422 25827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:06.813508 25827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:06.819864 25827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:06.819971 25827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:06.820057 25827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:06.820178 25827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:06.820242 25827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:06.828979 25827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:06.830044 25827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:06.831259 25827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:06.842223 25835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:06.842301 25832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:06.842249 25833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:06.843122 25827 server_base.cc:1034] running on GCE node
I20260925 15:12:06.843873 25827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:06.845098 25827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:06.846331 25827 hybrid_clock.cc:648] HybridClock initialized: now 1790349126846277 us; error 32 us; skew 500 ppm
I20260925 15:12:06.846611 25827 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:06 dist-test-slave-64fx krb5kdc[25814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349126, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:06.853884 25827 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:06.855124 25827 webserver.cc:460] Webserver started at http://127.0.0.1:41081/ using document root <none> and password file <none>
I20260925 15:12:06.855655 25827 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:06.855741 25827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:06.855949 25827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:06.858474 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "ef4104381520487db7961ce91d749815"
format_stamp: "Formatted at 2026-09-25 15:12:06 on dist-test-slave-64fx"
server_key: "b699463a36acdd591d34b798fc588ac3"
server_key_iv: "95584ea7293bc32d366ccf3aaffe35f4"
server_key_version: "encryptionkey@0"
I20260925 15:12:06.859093 25827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "ef4104381520487db7961ce91d749815"
format_stamp: "Formatted at 2026-09-25 15:12:06 on dist-test-slave-64fx"
server_key: "b699463a36acdd591d34b798fc588ac3"
server_key_iv: "95584ea7293bc32d366ccf3aaffe35f4"
server_key_version: "encryptionkey@0"
I20260925 15:12:06.864233 25827 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260925 15:12:06.867193 25842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:06.868361 25827 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:12:06.868556 25827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "ef4104381520487db7961ce91d749815"
format_stamp: "Formatted at 2026-09-25 15:12:06 on dist-test-slave-64fx"
server_key: "b699463a36acdd591d34b798fc588ac3"
server_key_iv: "95584ea7293bc32d366ccf3aaffe35f4"
server_key_version: "encryptionkey@0"
I20260925 15:12:06.868728 25827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:06.873909 25827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:06.877295 25827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:06.877547 25827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:06.894450 25827 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37875
I20260925 15:12:06.894521 25893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37875 every 8 connection(s)
I20260925 15:12:06.895699 25827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:06.899062 25894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:06.904239 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25827
I20260925 15:12:06.904405 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:06.904807 4919 external_mini_cluster.cc:1428] Setting key 9cb36c101c86f773371e9db2d672a0e9
I20260925 15:12:06.910413 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815: Bootstrap starting.
I20260925 15:12:06.913414 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:06.906603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59887 (local address 127.0.0.1:37875)
0925 15:12:06.907455 (+ 852us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:06.907470 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:06.907523 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:12:06.909810 (+ 2287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:06.909836 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:06.910230 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:06.910627 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:06.911532 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:06.911539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:06.911553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:06.911568 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:06.911629 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:06.911668 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:06.911923 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:06.912073 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 15:12:06.912271 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":370,"threads_started":1}
I20260925 15:12:06.914197 25894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:06.915282 25894 log.cc:824] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:06.916896 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:06.918072 25894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815: No bootstrap required, opened a new log
I20260925 15:12:06.923252 25897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:06.920882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42363 (local address 127.0.0.1:37875)
0925 15:12:06.921138 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:06.921144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:06.921481 (+ 337us) server_negotiation.cc:407] Connection header received
0925 15:12:06.921656 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:06.921660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:06.921699 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:06.921771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:06.922645 (+ 874us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:06.922649 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:06.922652 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:06.922664 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:06.922696 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:06.922713 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:06.922799 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:06.922911 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 15:12:06.922948 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:12:06.926385 25894 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260925 15:12:06.926712 25894 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:06.926802 25894 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef4104381520487db7961ce91d749815, State: Initialized, Role: FOLLOWER
I20260925 15:12:06.927269 25894 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260925 15:12:06.927484 25894 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:06.927640 25894 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:06.927798 25894 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:06.929865 25894 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260925 15:12:06.930302 25894 leader_election.cc:302] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef4104381520487db7961ce91d749815; no voters:
I20260925 15:12:06.930927 25894 leader_election.cc:288] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:06.931057 25904 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:06.931685 25904 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [term 1 LEADER]: Becoming Leader. State: Replica: ef4104381520487db7961ce91d749815, State: Running, Role: LEADER
I20260925 15:12:06.932406 25904 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } }
I20260925 15:12:06.932905 25894 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:06.935719 25906 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef4104381520487db7961ce91d749815. Latest consensus state: current_term: 1 leader_uuid: "ef4104381520487db7961ce91d749815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } } }
I20260925 15:12:06.935957 25906 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:06.935314 25905 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef4104381520487db7961ce91d749815" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef4104381520487db7961ce91d749815" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37875 } } }
I20260925 15:12:06.936394 25905 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:06.937486 25913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:06.941612 25913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:06.949363 25913 catalog_manager.cc:1347] Generated new cluster ID: 2bcae1cea970462489d3a8dd3ffbd72a
I20260925 15:12:06.949484 25913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:06.960181 25913 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:06.960599 25913 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:06.970289 25913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef4104381520487db7961ce91d749815: Generated new TSK 0
I20260925 15:12:06.971232 25913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:06.979838 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25827
W20260925 15:12:06.981321 25889 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59887
W20260925 15:12:06.981976 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25827: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:06.982093 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25827
2026-09-25T15:12:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1771 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:07 dist-test-slave-64fx krb5kdc[25925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:07 dist-test-slave-64fx krb5kdc[25925](info): set up 2 sockets
Sep 25 15:12:07 dist-test-slave-64fx krb5kdc[25925](info): commencing operation
krb5kdc: starting...
W20260925 15:12:08.049855 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[25925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:08.277189 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46115
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46115
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:08.599107 25938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:08.599383 25938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:08.599583 25938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:08.606804 25938 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:08.606910 25938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:08.607024 25938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:08.607149 25938 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:08.607235 25938 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:08.616736 25938 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46115
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46115
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:08.618007 25938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:08.619256 25938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:08.630483 25946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:08.630541 25943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:08.630563 25944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:08.631364 25938 server_base.cc:1034] running on GCE node
I20260925 15:12:08.632090 25938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:08.633373 25938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:08.634565 25938 hybrid_clock.cc:648] HybridClock initialized: now 1790349128634507 us; error 52 us; skew 500 ppm
I20260925 15:12:08.634923 25938 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[25925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349128, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:08.637989 25938 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:08.639266 25938 webserver.cc:460] Webserver started at http://127.0.0.1:35807/ using document root <none> and password file <none>
I20260925 15:12:08.639889 25938 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:08.639981 25938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:08.640213 25938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:08.642990 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "b654283576c8422db83a242e6f391e1b"
format_stamp: "Formatted at 2026-09-25 15:12:08 on dist-test-slave-64fx"
server_key: "78f4cd819bd30fd7ee92cb1de4439b60"
server_key_iv: "548345f475029d2c1661a59111ade607"
server_key_version: "encryptionkey@0"
I20260925 15:12:08.643671 25938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "b654283576c8422db83a242e6f391e1b"
format_stamp: "Formatted at 2026-09-25 15:12:08 on dist-test-slave-64fx"
server_key: "78f4cd819bd30fd7ee92cb1de4439b60"
server_key_iv: "548345f475029d2c1661a59111ade607"
server_key_version: "encryptionkey@0"
I20260925 15:12:08.648396 25938 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:08.651521 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:08.652747 25938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260925 15:12:08.652923 25938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "b654283576c8422db83a242e6f391e1b"
format_stamp: "Formatted at 2026-09-25 15:12:08 on dist-test-slave-64fx"
server_key: "78f4cd819bd30fd7ee92cb1de4439b60"
server_key_iv: "548345f475029d2c1661a59111ade607"
server_key_version: "encryptionkey@0"
I20260925 15:12:08.653100 25938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:08.680030 25938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:08.684376 25938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:08.684657 25938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:08.702565 25938 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46115
I20260925 15:12:08.702608 26004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46115 every 8 connection(s)
I20260925 15:12:08.703951 25938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:08.707504 26005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:08.710281 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 25938
I20260925 15:12:08.710445 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:08.710953 4919 external_mini_cluster.cc:1428] Setting key 52dee7abb1f925fdc4b8e137ce69b14a
I20260925 15:12:08.719417 26005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b: Bootstrap starting.
I20260925 15:12:08.723061 26005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:08.724406 26005 log.cc:824] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[25925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349128, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:12:08.727185 26005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b: No bootstrap required, opened a new log
I20260925 15:12:08.733690 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:08.712744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:46455 (local address 127.0.0.1:46115)
0925 15:12:08.713529 (+ 785us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:08.713542 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:08.713635 (+ 93us) server_negotiation.cc:407] Connection header received
0925 15:12:08.715269 (+ 1634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:08.715291 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:08.715704 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:08.716027 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:08.717778 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:08.719017 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:08.720663 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:08.721315 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:08.721447 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:08.726595 (+ 5148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:08.726602 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:08.726616 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:08.726654 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:08.729750 (+ 3096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:08.730492 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:08.730498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:08.730504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:08.730650 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:08.731103 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:08.731107 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:08.731108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:08.731582 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:08.731916 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:08.732355 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 15:12:08.732621 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":374,"threads_started":1}
I20260925 15:12:08.735030 26005 raft_consensus.cc:348] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260925 15:12:08.735394 26005 raft_consensus.cc:374] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:08.735530 26005 raft_consensus.cc:729] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b654283576c8422db83a242e6f391e1b, State: Initialized, Role: FOLLOWER
I20260925 15:12:08.736166 26005 consensus_queue.cc:260] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260925 15:12:08.736466 26005 raft_consensus.cc:388] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:08.736616 26005 raft_consensus.cc:482] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:08.736788 26005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:08.737869 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:08.739899 26005 raft_consensus.cc:504] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260925 15:12:08.740479 26005 leader_election.cc:302] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b654283576c8422db83a242e6f391e1b; no voters:
I20260925 15:12:08.741214 26005 leader_election.cc:288] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:08.741312 26010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:08.743453 26005 sys_catalog.cc:564] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:08.743527 26010 raft_consensus.cc:686] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [term 1 LEADER]: Becoming Leader. State: Replica: b654283576c8422db83a242e6f391e1b, State: Running, Role: LEADER
I20260925 15:12:08.744285 26010 consensus_queue.cc:237] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } }
I20260925 15:12:08.751187 26013 sys_catalog.cc:455] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b654283576c8422db83a242e6f391e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } } }
I20260925 15:12:08.751613 26013 sys_catalog.cc:458] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:08.752244 26012 sys_catalog.cc:455] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b654283576c8422db83a242e6f391e1b. Latest consensus state: current_term: 1 leader_uuid: "b654283576c8422db83a242e6f391e1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b654283576c8422db83a242e6f391e1b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46115 } } }
I20260925 15:12:08.752573 26012 sys_catalog.cc:458] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:08.757946 26018 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:08.763631 26018 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:08.777900 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:08.756087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:50993 (local address 127.0.0.1:46115)
0925 15:12:08.756319 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:08.756327 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:08.758868 (+ 2541us) server_negotiation.cc:407] Connection header received
0925 15:12:08.759346 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:08.759352 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:08.759432 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:08.759499 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:08.761972 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:08.763167 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:08.764982 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:08.765642 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:08.765737 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:08.768779 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:08.768784 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:08.768788 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:08.768822 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:08.774598 (+ 5776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:08.775552 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:08.775557 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:08.775561 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:08.775697 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:08.776273 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:08.776279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:08.776284 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:08.776701 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:08.776984 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:08.777335 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 15:12:08.777503 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 15:12:08.780359 26018 catalog_manager.cc:1347] Generated new cluster ID: 7a40cd9aa1744326920fcedd915fa0b8
I20260925 15:12:08.780455 26018 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:08.790207 26018 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:08.791711 26018 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:08.800976 26018 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b654283576c8422db83a242e6f391e1b: Generated new TSK 0
I20260925 15:12:08.801939 26018 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:08.809343 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 25938
I20260925 15:12:08.810424 26000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:08.897828 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 25938
W20260925 15:12:08.914536 25923 connection.cc:535] client connection to 127.0.0.1:46115 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46115): Connection reset by peer (error 104)
2026-09-25T15:12:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1915 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[26038](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[26038](info): set up 2 sockets
Sep 25 15:12:08 dist-test-slave-64fx krb5kdc[26038](info): commencing operation
krb5kdc: starting...
W20260925 15:12:09.998167 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.006s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:10 dist-test-slave-64fx krb5kdc[26038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:10.236323 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:10.558636 26051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:10.558913 26051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:10.559079 26051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:10.566110 26051 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:10.566254 26051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:10.566365 26051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:10.566468 26051 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:10.566555 26051 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:10.576061 26051 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37987
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:10.577270 26051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:10.578620 26051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:10.590415 26056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:10.590477 26057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:10.590448 26059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:10.591005 26051 server_base.cc:1034] running on GCE node
I20260925 15:12:10.591902 26051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:10.593288 26051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:10.594518 26051 hybrid_clock.cc:648] HybridClock initialized: now 1790349130594473 us; error 54 us; skew 500 ppm
I20260925 15:12:10.594866 26051 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:10 dist-test-slave-64fx krb5kdc[26038](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349130, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:10.597988 26051 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:10.599318 26051 webserver.cc:460] Webserver started at http://127.0.0.1:42463/ using document root <none> and password file <none>
I20260925 15:12:10.599964 26051 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:10.600062 26051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:10.600298 26051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:10.603190 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d74ffd9725f54d67b081aef69c65bc3d"
format_stamp: "Formatted at 2026-09-25 15:12:10 on dist-test-slave-64fx"
server_key: "4d2a787b753771a0b8249b39e24afb24"
server_key_iv: "5aaafa824b23ee9ad16f4f382210280a"
server_key_version: "encryptionkey@0"
I20260925 15:12:10.603893 26051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d74ffd9725f54d67b081aef69c65bc3d"
format_stamp: "Formatted at 2026-09-25 15:12:10 on dist-test-slave-64fx"
server_key: "4d2a787b753771a0b8249b39e24afb24"
server_key_iv: "5aaafa824b23ee9ad16f4f382210280a"
server_key_version: "encryptionkey@0"
I20260925 15:12:10.608774 26051 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:12:10.612090 26066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:10.613373 26051 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:10.613543 26051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d74ffd9725f54d67b081aef69c65bc3d"
format_stamp: "Formatted at 2026-09-25 15:12:10 on dist-test-slave-64fx"
server_key: "4d2a787b753771a0b8249b39e24afb24"
server_key_iv: "5aaafa824b23ee9ad16f4f382210280a"
server_key_version: "encryptionkey@0"
I20260925 15:12:10.613819 26051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:10.637961 26051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:10.642453 26051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:10.642751 26051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:10.661208 26051 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37987
I20260925 15:12:10.661252 26117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37987 every 8 connection(s)
I20260925 15:12:10.662632 26051 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:10.666505 26118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:10.670169 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26051
I20260925 15:12:10.670367 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:10.670905 4919 external_mini_cluster.cc:1428] Setting key 670052515f1d5b8a920eb113c860d10e
I20260925 15:12:10.678540 26118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d: Bootstrap starting.
I20260925 15:12:10.681541 26118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:10.682691 26118 log.cc:824] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:10.683807 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:10.672805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:52199 (local address 127.0.0.1:37987)
0925 15:12:10.673499 (+ 694us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:10.673511 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:10.673598 (+ 87us) server_negotiation.cc:407] Connection header received
0925 15:12:10.675229 (+ 1631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:10.675251 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:10.675658 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:10.676039 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:10.676600 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:10.677937 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:10.679351 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:10.680029 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:10.680250 (+ 221us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:10.681476 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:10.681486 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:10.681499 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:10.682149 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:10.682222 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:10.682246 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:10.682407 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:10.682680 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 15:12:10.682883 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":295,"threads_started":1}
I20260925 15:12:10.685428 26118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d: No bootstrap required, opened a new log
I20260925 15:12:10.688045 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:10.696302 26118 raft_consensus.cc:348] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } }
I20260925 15:12:10.696666 26118 raft_consensus.cc:374] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:10.696761 26118 raft_consensus.cc:729] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d74ffd9725f54d67b081aef69c65bc3d, State: Initialized, Role: FOLLOWER
I20260925 15:12:10.697461 26118 consensus_queue.cc:260] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } }
I20260925 15:12:10.697777 26118 raft_consensus.cc:388] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:10.697930 26118 raft_consensus.cc:482] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:10.698120 26118 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:10.699800 26121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:10.693125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:44087 (local address 127.0.0.1:37987)
0925 15:12:10.693311 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:10.693319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:10.693734 (+ 415us) server_negotiation.cc:407] Connection header received
0925 15:12:10.693878 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:10.693883 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:10.693961 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:10.694077 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:10.694545 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:10.695703 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:10.697067 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:10.697776 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:10.697907 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:10.699074 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:10.699080 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:10.699085 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:10.699109 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:10.699156 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:10.699178 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:10.699292 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:10.699423 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:12:10.699487 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:12:10.700763 26118 raft_consensus.cc:504] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } }
I20260925 15:12:10.701270 26118 leader_election.cc:302] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d74ffd9725f54d67b081aef69c65bc3d; no voters:
I20260925 15:12:10.701822 26118 leader_election.cc:288] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:10.701902 26128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:10.702596 26128 raft_consensus.cc:686] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [term 1 LEADER]: Becoming Leader. State: Replica: d74ffd9725f54d67b081aef69c65bc3d, State: Running, Role: LEADER
I20260925 15:12:10.703099 26128 consensus_queue.cc:237] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } }
I20260925 15:12:10.703788 26118 sys_catalog.cc:564] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:10.706054 26130 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d74ffd9725f54d67b081aef69c65bc3d. Latest consensus state: current_term: 1 leader_uuid: "d74ffd9725f54d67b081aef69c65bc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } } }
I20260925 15:12:10.706629 26130 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:10.706689 26129 sys_catalog.cc:455] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d74ffd9725f54d67b081aef69c65bc3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d74ffd9725f54d67b081aef69c65bc3d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37987 } } }
I20260925 15:12:10.706985 26129 sys_catalog.cc:458] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:10.708866 26136 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:10.713716 26136 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:10.723438 26136 catalog_manager.cc:1347] Generated new cluster ID: b4e22d9284a3414fa9fc4da41478c0bd
I20260925 15:12:10.723541 26136 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:10.747093 26136 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:10.748927 26136 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:10.756959 26136 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d74ffd9725f54d67b081aef69c65bc3d: Generated new TSK 0
I20260925 15:12:10.758035 26136 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:10.766722 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26051
W20260925 15:12:10.767949 26113 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.250:52199
W20260925 15:12:10.768716 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26051: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:10.768869 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26051
W20260925 15:12:10.785786 26036 connection.cc:535] client connection to 127.0.0.1:37987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37987): Connection reset by peer (error 104)
2026-09-25T15:12:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1871 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:10 dist-test-slave-64fx krb5kdc[26149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:10 dist-test-slave-64fx krb5kdc[26149](info): set up 2 sockets
Sep 25 15:12:10 dist-test-slave-64fx krb5kdc[26149](info): commencing operation
krb5kdc: starting...
W20260925 15:12:11.849897 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:11 dist-test-slave-64fx krb5kdc[26149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349131, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:12.076224 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36245
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:12.385010 26162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:12.385264 26162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:12.385345 26162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:12.391687 26162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:12.391774 26162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:12.391817 26162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:12.391863 26162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:12.391896 26162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:12.404239 26162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:12.405336 26162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:12.406597 26162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:12.417481 26168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:12.417434 26170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:12.417536 26167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:12.418391 26162 server_base.cc:1034] running on GCE node
I20260925 15:12:12.419082 26162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:12.420303 26162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:12.421523 26162 hybrid_clock.cc:648] HybridClock initialized: now 1790349132421490 us; error 46 us; skew 500 ppm
I20260925 15:12:12.421926 26162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:12 dist-test-slave-64fx krb5kdc[26149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349132, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:12.429657 26162 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:12.431044 26162 webserver.cc:460] Webserver started at http://127.0.0.1:39637/ using document root <none> and password file <none>
I20260925 15:12:12.431638 26162 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:12.431736 26162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:12.431955 26162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:12.434780 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d8068bb2074f49f79db8605e5057d67c"
format_stamp: "Formatted at 2026-09-25 15:12:12 on dist-test-slave-64fx"
server_key: "f7a64b9392eb989672e3ebd1fb2f272f"
server_key_iv: "7a2d663da1cc98859a2d3458cd78a575"
server_key_version: "encryptionkey@0"
I20260925 15:12:12.435461 26162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d8068bb2074f49f79db8605e5057d67c"
format_stamp: "Formatted at 2026-09-25 15:12:12 on dist-test-slave-64fx"
server_key: "f7a64b9392eb989672e3ebd1fb2f272f"
server_key_iv: "7a2d663da1cc98859a2d3458cd78a575"
server_key_version: "encryptionkey@0"
I20260925 15:12:12.440070 26162 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:12.443251 26177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:12.444480 26162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:12:12.444651 26162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d8068bb2074f49f79db8605e5057d67c"
format_stamp: "Formatted at 2026-09-25 15:12:12 on dist-test-slave-64fx"
server_key: "f7a64b9392eb989672e3ebd1fb2f272f"
server_key_iv: "7a2d663da1cc98859a2d3458cd78a575"
server_key_version: "encryptionkey@0"
I20260925 15:12:12.444824 26162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:12.450290 26162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:12.453984 26162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:12.454282 26162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:12.471982 26162 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36245
I20260925 15:12:12.472072 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36245 every 8 connection(s)
I20260925 15:12:12.473369 26162 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:12.475359 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26162
I20260925 15:12:12.475526 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:12.475966 4919 external_mini_cluster.cc:1428] Setting key dd8c61b9b8c1b2bc58c9c1fbd1050d05
I20260925 15:12:12.477169 26229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:12.483626 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:12.477734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:54237 (local address 127.0.0.1:36245)
0925 15:12:12.478477 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:12.478487 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:12.478527 (+ 40us) server_negotiation.cc:407] Connection header received
0925 15:12:12.480178 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:12.480197 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:12.480527 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:12.480838 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:12.482051 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:12.482057 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:12.482069 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:12.482084 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:12.482168 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:12.482192 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:12.482372 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:12.482490 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:12:12.482721 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":353,"threads_started":1}
I20260925 15:12:12.487217 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:12.489840 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c: Bootstrap starting.
I20260925 15:12:12.493041 26229 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:12.494022 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:12.491579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:52287 (local address 127.0.0.1:36245)
0925 15:12:12.491727 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:12.491731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:12.492119 (+ 388us) server_negotiation.cc:407] Connection header received
0925 15:12:12.492302 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:12.492313 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:12.492356 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:12.492474 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:12.493377 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:12.493383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:12.493388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:12.493416 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:12.493459 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:12.493475 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:12.493611 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:12.493707 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 15:12:12.493764 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:12:12.494241 26229 log.cc:824] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:12.496724 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c: No bootstrap required, opened a new log
I20260925 15:12:12.503846 26229 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 15:12:12.504186 26229 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:12.504297 26229 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8068bb2074f49f79db8605e5057d67c, State: Initialized, Role: FOLLOWER
I20260925 15:12:12.504822 26229 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 15:12:12.505038 26229 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:12.505184 26229 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:12.505358 26229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:12.507643 26229 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 15:12:12.508076 26229 leader_election.cc:302] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8068bb2074f49f79db8605e5057d67c; no voters:
I20260925 15:12:12.508605 26229 leader_election.cc:288] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:12.508744 26239 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:12.509356 26239 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [term 1 LEADER]: Becoming Leader. State: Replica: d8068bb2074f49f79db8605e5057d67c, State: Running, Role: LEADER
I20260925 15:12:12.509905 26239 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } }
I20260925 15:12:12.510463 26229 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:12.512444 26240 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8068bb2074f49f79db8605e5057d67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260925 15:12:12.512835 26240 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:12.513309 26241 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8068bb2074f49f79db8605e5057d67c. Latest consensus state: current_term: 1 leader_uuid: "d8068bb2074f49f79db8605e5057d67c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8068bb2074f49f79db8605e5057d67c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36245 } } }
I20260925 15:12:12.513656 26241 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:12.517510 26247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:12.522902 26247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:12.530717 26247 catalog_manager.cc:1347] Generated new cluster ID: fc34376f09044c17a9f1565677596b55
I20260925 15:12:12.530808 26247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:12.549011 26247 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:12.549475 26247 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:12.559031 26247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8068bb2074f49f79db8605e5057d67c: Generated new TSK 0
I20260925 15:12:12.559710 26247 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:12.573742 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26162
W20260925 15:12:12.574937 26224 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.250:54237
W20260925 15:12:12.575557 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:12.575703 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26162
2026-09-25T15:12:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1805 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:12 dist-test-slave-64fx krb5kdc[26261](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:12 dist-test-slave-64fx krb5kdc[26261](info): set up 2 sockets
Sep 25 15:12:12 dist-test-slave-64fx krb5kdc[26261](info): commencing operation
krb5kdc: starting...
W20260925 15:12:13.633939 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.008s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:13 dist-test-slave-64fx krb5kdc[26261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:13.854740 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:14.159896 26274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:14.160141 26274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:14.160219 26274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:14.167079 26274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:14.167194 26274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:14.167317 26274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:14.167455 26274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:14.167538 26274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:14.176340 26274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:14.177475 26274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:14.178640 26274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:14.189611 26279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:14.189644 26280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:14.189838 26282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:14.190374 26274 server_base.cc:1034] running on GCE node
I20260925 15:12:14.191013 26274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:14.192225 26274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:14.193449 26274 hybrid_clock.cc:648] HybridClock initialized: now 1790349134193416 us; error 41 us; skew 500 ppm
I20260925 15:12:14.193877 26274 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:14 dist-test-slave-64fx krb5kdc[26261](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:14.196625 26274 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:14.197930 26274 webserver.cc:460] Webserver started at http://127.0.0.1:46279/ using document root <none> and password file <none>
I20260925 15:12:14.198529 26274 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:14.198664 26274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:14.198931 26274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:14.201534 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "23366699003e4bc7b2928f3a12d420b8"
format_stamp: "Formatted at 2026-09-25 15:12:14 on dist-test-slave-64fx"
server_key: "fa98d6bd7ee587afd59e3e9a7124d27a"
server_key_iv: "20fc89bb34bf1bad9291350a04590c6c"
server_key_version: "encryptionkey@0"
I20260925 15:12:14.202316 26274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "23366699003e4bc7b2928f3a12d420b8"
format_stamp: "Formatted at 2026-09-25 15:12:14 on dist-test-slave-64fx"
server_key: "fa98d6bd7ee587afd59e3e9a7124d27a"
server_key_iv: "20fc89bb34bf1bad9291350a04590c6c"
server_key_version: "encryptionkey@0"
I20260925 15:12:14.206719 26274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 15:12:14.209753 26289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:14.210907 26274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:12:14.211081 26274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "23366699003e4bc7b2928f3a12d420b8"
format_stamp: "Formatted at 2026-09-25 15:12:14 on dist-test-slave-64fx"
server_key: "fa98d6bd7ee587afd59e3e9a7124d27a"
server_key_iv: "20fc89bb34bf1bad9291350a04590c6c"
server_key_version: "encryptionkey@0"
I20260925 15:12:14.211261 26274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:14.229424 26274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:14.232923 26274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:14.233218 26274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:14.250634 26274 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45135
I20260925 15:12:14.250669 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45135 every 8 connection(s)
I20260925 15:12:14.251861 26274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:14.253772 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26274
I20260925 15:12:14.253947 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:14.254513 4919 external_mini_cluster.cc:1428] Setting key d0b2fc9754cfad85ffb414b05b0ef850
I20260925 15:12:14.255941 26341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:14.269903 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8: Bootstrap starting.
Sep 25 15:12:14 dist-test-slave-64fx krb5kdc[26261](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:12:14.273427 26341 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:14.274811 26341 log.cc:824] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:14.278091 26341 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8: No bootstrap required, opened a new log
I20260925 15:12:14.279996 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:14.256353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:60121 (local address 127.0.0.1:45135)
0925 15:12:14.259625 (+ 3272us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:14.259640 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:14.259684 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:12:14.261248 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:14.261278 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:14.261797 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:14.262218 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:14.263455 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:14.265351 (+ 1896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:14.266591 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:14.267632 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:14.267815 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:14.271589 (+ 3774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:14.271599 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:14.271619 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:14.271659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:14.275468 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:14.276342 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:14.276350 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:14.276357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:14.276558 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:14.276983 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:14.276988 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:14.276992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:14.277682 (+ 690us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:14.278011 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:14.278344 (+ 333us) server_negotiation.cc:299] Negotiation successful
0925 15:12:14.278675 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3052,"thread_start_us":2809,"threads_started":1}
I20260925 15:12:14.283877 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:14.287065 26341 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260925 15:12:14.287431 26341 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:14.287534 26341 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23366699003e4bc7b2928f3a12d420b8, State: Initialized, Role: FOLLOWER
I20260925 15:12:14.288082 26341 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260925 15:12:14.288389 26341 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:14.288552 26341 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:14.288826 26341 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:14.292098 26341 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260925 15:12:14.292695 26341 leader_election.cc:302] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23366699003e4bc7b2928f3a12d420b8; no voters:
I20260925 15:12:14.293313 26341 leader_election.cc:288] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:14.293457 26351 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:14.294773 26351 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [term 1 LEADER]: Becoming Leader. State: Replica: 23366699003e4bc7b2928f3a12d420b8, State: Running, Role: LEADER
I20260925 15:12:14.295393 26351 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260925 15:12:14.297953 26341 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:14.298662 26352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23366699003e4bc7b2928f3a12d420b8. Latest consensus state: current_term: 1 leader_uuid: "23366699003e4bc7b2928f3a12d420b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260925 15:12:14.298686 26353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23366699003e4bc7b2928f3a12d420b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23366699003e4bc7b2928f3a12d420b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260925 15:12:14.299101 26352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:14.299386 26353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:14.300052 26356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:14.307917 26344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:14.288454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:35577 (local address 127.0.0.1:45135)
0925 15:12:14.288663 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:14.288671 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:14.289096 (+ 425us) server_negotiation.cc:407] Connection header received
0925 15:12:14.289309 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:14.289312 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:14.289372 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:14.289448 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:14.291316 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:14.292087 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:14.293929 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:14.294950 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:14.295078 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:14.298287 (+ 3209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:14.298294 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:14.298298 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:14.298335 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:14.302845 (+ 4510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:14.304095 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:14.304103 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:14.304123 (+ 20us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:14.304316 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:14.305021 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:14.305028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:14.305034 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:14.305540 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:14.306955 (+ 1415us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:14.307226 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 15:12:14.307450 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:12:14.317384 26356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:14.326457 26356 catalog_manager.cc:1347] Generated new cluster ID: d8359a97eda940fbb14c06d74f2a8127
I20260925 15:12:14.326567 26356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:14.343559 26356 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:14.344782 26356 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:14.351394 26356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23366699003e4bc7b2928f3a12d420b8: Generated new TSK 0
I20260925 15:12:14.352090 26356 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:14.361392 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26274
I20260925 15:12:14.363319 26336 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:14.456341 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26274
W20260925 15:12:14.469807 26259 connection.cc:535] client connection to 127.0.0.1:45135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45135): Connection reset by peer (error 104)
2026-09-25T15:12:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1878 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:14 dist-test-slave-64fx krb5kdc[26373](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:14 dist-test-slave-64fx krb5kdc[26373](info): set up 2 sockets
Sep 25 15:12:14 dist-test-slave-64fx krb5kdc[26373](info): commencing operation
krb5kdc: starting...
W20260925 15:12:15.497843 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:15 dist-test-slave-64fx krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349135, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:15.732857 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44937
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44937
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:16.041196 26386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:16.041450 26386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:16.041652 26386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:16.048476 26386 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:16.048565 26386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:16.048614 26386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:16.048689 26386 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:16.048754 26386 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:16.058746 26386 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44937
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:16.059919 26386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:16.061189 26386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:16.072162 26391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:16.072198 26394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:16.072156 26392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:16.072912 26386 server_base.cc:1034] running on GCE node
I20260925 15:12:16.073637 26386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:16.074909 26386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:16.076112 26386 hybrid_clock.cc:648] HybridClock initialized: now 1790349136076074 us; error 31 us; skew 500 ppm
I20260925 15:12:16.076433 26386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:16 dist-test-slave-64fx krb5kdc[26373](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349136, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:16.079429 26386 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:16.080796 26386 webserver.cc:460] Webserver started at http://127.0.0.1:45675/ using document root <none> and password file <none>
I20260925 15:12:16.081459 26386 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:16.081637 26386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:16.081931 26386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:16.084709 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "96b3c551a4c242739065eab47f237d0f"
format_stamp: "Formatted at 2026-09-25 15:12:16 on dist-test-slave-64fx"
server_key: "55b1d851ca1fce5599f02dbfd70b9ef4"
server_key_iv: "352b20dc5f9d7e97b0ec572b744e5f28"
server_key_version: "encryptionkey@0"
I20260925 15:12:16.085453 26386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "96b3c551a4c242739065eab47f237d0f"
format_stamp: "Formatted at 2026-09-25 15:12:16 on dist-test-slave-64fx"
server_key: "55b1d851ca1fce5599f02dbfd70b9ef4"
server_key_iv: "352b20dc5f9d7e97b0ec572b744e5f28"
server_key_version: "encryptionkey@0"
I20260925 15:12:16.090021 26386 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 15:12:16.093282 26401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:16.094589 26386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:16.094766 26386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "96b3c551a4c242739065eab47f237d0f"
format_stamp: "Formatted at 2026-09-25 15:12:16 on dist-test-slave-64fx"
server_key: "55b1d851ca1fce5599f02dbfd70b9ef4"
server_key_iv: "352b20dc5f9d7e97b0ec572b744e5f28"
server_key_version: "encryptionkey@0"
I20260925 15:12:16.094947 26386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:16.121654 26386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:16.125841 26386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:16.126148 26386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:16.143667 26386 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44937
I20260925 15:12:16.143708 26452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44937 every 8 connection(s)
I20260925 15:12:16.144982 26386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:16.148530 26453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:16.155364 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26386
I20260925 15:12:16.155529 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:16.155998 4919 external_mini_cluster.cc:1428] Setting key 7f9bf27be035e47fb3da0795fd21b4de
I20260925 15:12:16.159356 26453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f: Bootstrap starting.
I20260925 15:12:16.162381 26453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:16.163306 26453 log.cc:824] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:16.166326 26453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f: No bootstrap required, opened a new log
I20260925 15:12:16.168705 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:16.157789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:41393 (local address 127.0.0.1:44937)
0925 15:12:16.158471 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:16.158498 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:16.158540 (+ 42us) server_negotiation.cc:407] Connection header received
0925 15:12:16.160111 (+ 1571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:16.160132 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:16.160474 (+ 342us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:16.160850 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:16.161397 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:16.162752 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:16.164282 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:16.165252 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:16.165435 (+ 183us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:16.166597 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:16.166604 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:16.166630 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:16.167141 (+ 511us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:16.167203 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:16.167225 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:16.167392 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:16.167655 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 15:12:16.167801 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":295,"threads_started":1}
I20260925 15:12:16.172026 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:16.173825 26453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:12:16.174172 26453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:16.174275 26453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96b3c551a4c242739065eab47f237d0f, State: Initialized, Role: FOLLOWER
I20260925 15:12:16.174854 26453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:12:16.175112 26453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:16.175256 26453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:16.175427 26453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:16.178123 26453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:12:16.178628 26453 leader_election.cc:302] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 96b3c551a4c242739065eab47f237d0f; no voters:
I20260925 15:12:16.179268 26453 leader_election.cc:288] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:16.179354 26463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:16.181577 26453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:16.182521 26456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:16.176325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:49693 (local address 127.0.0.1:44937)
0925 15:12:16.176469 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:16.176478 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:16.176738 (+ 260us) server_negotiation.cc:407] Connection header received
0925 15:12:16.176831 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:16.176837 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:16.176910 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:16.177010 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:16.177409 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:16.178388 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:16.179828 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:16.180392 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:16.180461 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:16.181673 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:16.181680 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:16.181687 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:16.181709 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:16.181786 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:16.181803 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:16.181921 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:16.182108 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:12:16.182168 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:12:16.186627 26463 raft_consensus.cc:686] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [term 1 LEADER]: Becoming Leader. State: Replica: 96b3c551a4c242739065eab47f237d0f, State: Running, Role: LEADER
I20260925 15:12:16.187224 26463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } }
I20260925 15:12:16.190227 26464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "96b3c551a4c242739065eab47f237d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } } }
I20260925 15:12:16.190603 26464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:16.190536 26465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96b3c551a4c242739065eab47f237d0f. Latest consensus state: current_term: 1 leader_uuid: "96b3c551a4c242739065eab47f237d0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "96b3c551a4c242739065eab47f237d0f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44937 } } }
I20260925 15:12:16.190762 26465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:16.192358 26473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:16.196848 26473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:16.205765 26473 catalog_manager.cc:1347] Generated new cluster ID: a177a0e505ac45f9aa87cc76661a20d2
I20260925 15:12:16.205888 26473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:16.220137 26473 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:16.221318 26473 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:16.227173 26473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 96b3c551a4c242739065eab47f237d0f: Generated new TSK 0
I20260925 15:12:16.227861 26473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:16.235296 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26386
W20260925 15:12:16.236702 26448 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.250:41393
W20260925 15:12:16.237432 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26386: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:16.237586 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26386
W20260925 15:12:16.254073 26371 connection.cc:535] client connection to 127.0.0.1:44937 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44937): Connection reset by peer (error 104)
2026-09-25T15:12:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1783 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:16 dist-test-slave-64fx krb5kdc[26484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:16 dist-test-slave-64fx krb5kdc[26484](info): set up 2 sockets
Sep 25 15:12:16 dist-test-slave-64fx krb5kdc[26484](info): commencing operation
krb5kdc: starting...
W20260925 15:12:17.314023 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:17 dist-test-slave-64fx krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349137, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:12:17.538733 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45747
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45747
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:17.848169 26497 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:17.848418 26497 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:17.848506 26497 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:17.855358 26497 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:17.855454 26497 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:17.855496 26497 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:17.855549 26497 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:17.855633 26497 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:17.864787 26497 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45747
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45747
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:17.866012 26497 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:17.867265 26497 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:17.878563 26505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:17.878624 26503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:17.879067 26502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:17.879228 26497 server_base.cc:1034] running on GCE node
I20260925 15:12:17.879958 26497 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:17.881247 26497 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:17.882465 26497 hybrid_clock.cc:648] HybridClock initialized: now 1790349137882421 us; error 53 us; skew 500 ppm
I20260925 15:12:17.882803 26497 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:17 dist-test-slave-64fx krb5kdc[26484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349137, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:17.893205 26497 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:12:17.894702 26497 webserver.cc:460] Webserver started at http://127.0.0.1:35941/ using document root <none> and password file <none>
I20260925 15:12:17.895391 26497 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:17.895537 26497 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:17.895831 26497 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:17.898798 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "fd30ca7c3da048c4a6cc4d7956b88751"
format_stamp: "Formatted at 2026-09-25 15:12:17 on dist-test-slave-64fx"
server_key: "7444cb5d06bd37c7e737d46aecb9cead"
server_key_iv: "4d81dcebc31a26b1753190e9fe286edb"
server_key_version: "encryptionkey@0"
I20260925 15:12:17.899544 26497 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "fd30ca7c3da048c4a6cc4d7956b88751"
format_stamp: "Formatted at 2026-09-25 15:12:17 on dist-test-slave-64fx"
server_key: "7444cb5d06bd37c7e737d46aecb9cead"
server_key_iv: "4d81dcebc31a26b1753190e9fe286edb"
server_key_version: "encryptionkey@0"
I20260925 15:12:17.904484 26497 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 15:12:17.907850 26512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:17.909113 26497 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 15:12:17.909287 26497 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "fd30ca7c3da048c4a6cc4d7956b88751"
format_stamp: "Formatted at 2026-09-25 15:12:17 on dist-test-slave-64fx"
server_key: "7444cb5d06bd37c7e737d46aecb9cead"
server_key_iv: "4d81dcebc31a26b1753190e9fe286edb"
server_key_version: "encryptionkey@0"
I20260925 15:12:17.909500 26497 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:17.915131 26497 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:17.918722 26497 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:17.919001 26497 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:17.936415 26497 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45747
I20260925 15:12:17.936458 26563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45747 every 8 connection(s)
I20260925 15:12:17.937965 26497 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:17.941174 26564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:17.947327 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26497
I20260925 15:12:17.947499 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:17.947902 4919 external_mini_cluster.cc:1428] Setting key 5e6ee1772c971dedcd1dfe40c693e487
I20260925 15:12:17.951350 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751: Bootstrap starting.
I20260925 15:12:17.955268 26564 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:17.956553 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:17.949716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:60259 (local address 127.0.0.1:45747)
0925 15:12:17.950467 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:17.950482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:17.950537 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:12:17.952845 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:17.952873 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:17.953314 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:17.953776 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:17.954723 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:17.954732 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:17.954748 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:17.954771 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:17.954834 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:17.954864 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:17.955043 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:17.955174 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:12:17.955365 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":294,"threads_started":1}
I20260925 15:12:17.956552 26564 log.cc:824] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:17.959741 26564 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751: No bootstrap required, opened a new log
I20260925 15:12:17.960443 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:17.967247 26567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:17.964739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.250:32823 (local address 127.0.0.1:45747)
0925 15:12:17.964922 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:17.964928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:17.965295 (+ 367us) server_negotiation.cc:407] Connection header received
0925 15:12:17.965491 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:17.965501 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:17.965546 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:17.965700 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:17.966590 (+ 890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:17.966604 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:17.966610 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:17.966632 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:17.966668 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:17.966681 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:17.966812 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:17.966916 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 15:12:17.966982 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 15:12:17.967924 26564 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } }
I20260925 15:12:17.968300 26564 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:17.968406 26564 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd30ca7c3da048c4a6cc4d7956b88751, State: Initialized, Role: FOLLOWER
I20260925 15:12:17.968959 26564 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } }
I20260925 15:12:17.969174 26564 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:17.969266 26564 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:17.969378 26564 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:17.971582 26564 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } }
I20260925 15:12:17.972015 26564 leader_election.cc:302] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd30ca7c3da048c4a6cc4d7956b88751; no voters:
I20260925 15:12:17.972545 26564 leader_election.cc:288] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:17.972667 26574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:17.973423 26574 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [term 1 LEADER]: Becoming Leader. State: Replica: fd30ca7c3da048c4a6cc4d7956b88751, State: Running, Role: LEADER
I20260925 15:12:17.974116 26574 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } }
I20260925 15:12:17.974887 26564 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:17.976928 26575 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } } }
I20260925 15:12:17.977244 26575 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:17.977900 26576 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd30ca7c3da048c4a6cc4d7956b88751. Latest consensus state: current_term: 1 leader_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd30ca7c3da048c4a6cc4d7956b88751" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45747 } } }
I20260925 15:12:17.978190 26576 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:17.979005 26580 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:17.984465 26580 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:17.992956 26580 catalog_manager.cc:1347] Generated new cluster ID: d77d85559b6e4084aa4e2fd00030649e
I20260925 15:12:17.993067 26580 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:18.014810 26580 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:18.015196 26580 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:18.022997 26580 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd30ca7c3da048c4a6cc4d7956b88751: Generated new TSK 0
I20260925 15:12:18.023677 26580 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:18.027901 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26497
W20260925 15:12:18.028803 26559 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.250:60259
W20260925 15:12:18.029441 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26497: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:18.029621 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26497
2026-09-25T15:12:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1796 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:18 dist-test-slave-64fx krb5kdc[26595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:18 dist-test-slave-64fx krb5kdc[26595](info): set up 2 sockets
Sep 25 15:12:18 dist-test-slave-64fx krb5kdc[26595](info): commencing operation
krb5kdc: starting...
W20260925 15:12:19.089769 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:19 dist-test-slave-64fx krb5kdc[26595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:19.320945 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:43005
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44815
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:43005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:19.650808 26608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:19.651072 26608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:19.651230 26608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:19.658381 26608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:19.658504 26608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:19.658597 26608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:19.658701 26608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:19.658784 26608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:19.669067 26608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44815
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:43005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:43005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:19.670284 26608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:19.671521 26608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:19.683115 26613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:19.683127 26614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:19.683616 26616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:19.684146 26608 server_base.cc:1034] running on GCE node
I20260925 15:12:19.684899 26608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:19.686239 26608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:19.687435 26608 hybrid_clock.cc:648] HybridClock initialized: now 1790349139687386 us; error 48 us; skew 500 ppm
I20260925 15:12:19.687734 26608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:19 dist-test-slave-64fx krb5kdc[26595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349139, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:19.690946 26608 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:19.692379 26608 webserver.cc:460] Webserver started at http://127.4.205.254:42111/ using document root <none> and password file <none>
I20260925 15:12:19.693037 26608 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:19.693166 26608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:19.693465 26608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:19.696311 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "7f43e28390a64442991c844cce0013b7"
format_stamp: "Formatted at 2026-09-25 15:12:19 on dist-test-slave-64fx"
server_key: "7e5b5d41033edee6ec6a1b77d8f768a1"
server_key_iv: "50481119f66acef9b6ccd72a4231fcc8"
server_key_version: "encryptionkey@0"
I20260925 15:12:19.697050 26608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "7f43e28390a64442991c844cce0013b7"
format_stamp: "Formatted at 2026-09-25 15:12:19 on dist-test-slave-64fx"
server_key: "7e5b5d41033edee6ec6a1b77d8f768a1"
server_key_iv: "50481119f66acef9b6ccd72a4231fcc8"
server_key_version: "encryptionkey@0"
I20260925 15:12:19.701993 26608 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260925 15:12:19.705276 26623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:19.706602 26608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:19.706784 26608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "7f43e28390a64442991c844cce0013b7"
format_stamp: "Formatted at 2026-09-25 15:12:19 on dist-test-slave-64fx"
server_key: "7e5b5d41033edee6ec6a1b77d8f768a1"
server_key_iv: "50481119f66acef9b6ccd72a4231fcc8"
server_key_version: "encryptionkey@0"
I20260925 15:12:19.706969 26608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:19.727413 26608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:19.731114 26608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:19.731381 26608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:19.749043 26608 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:43005
I20260925 15:12:19.749055 26674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:43005 every 8 connection(s)
I20260925 15:12:19.750386 26608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:19.752133 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26608
I20260925 15:12:19.752300 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:19.752847 4919 external_mini_cluster.cc:1428] Setting key 5471776b2914f4ccc640315df2dd428b
I20260925 15:12:19.754535 26675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:19.765513 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7: Bootstrap starting.
Sep 25 15:12:19 dist-test-slave-64fx krb5kdc[26595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349139, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:19.769105 26675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:19.770236 26675 log.cc:824] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:19.773362 26675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7: No bootstrap required, opened a new log
I20260925 15:12:19.774482 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:19.754571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42093 (local address 127.4.205.254:43005)
0925 15:12:19.755264 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:19.755275 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:19.755312 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:12:19.756876 (+ 1564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:19.756895 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:19.757257 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:19.757645 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:19.759078 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:19.760491 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:19.761755 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:19.762442 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:19.762645 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:19.767072 (+ 4427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:19.767083 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:19.767097 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:19.767136 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:19.770402 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:19.771280 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:19.771294 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:19.771301 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:19.771529 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:19.771979 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:19.771982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:19.771984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:19.772447 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:19.772705 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:19.773189 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 15:12:19.773503 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":317,"threads_started":1}
I20260925 15:12:19.778366 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:19.781462 26675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } }
I20260925 15:12:19.781893 26675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:19.782008 26675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f43e28390a64442991c844cce0013b7, State: Initialized, Role: FOLLOWER
I20260925 15:12:19.782646 26675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } }
I20260925 15:12:19.783013 26675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:19.783155 26675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:19.783346 26675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:19.786366 26675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } }
I20260925 15:12:19.786886 26675 leader_election.cc:302] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f43e28390a64442991c844cce0013b7; no voters:
I20260925 15:12:19.787497 26675 leader_election.cc:288] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:19.787626 26685 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:19.789618 26685 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7f43e28390a64442991c844cce0013b7, State: Running, Role: LEADER
I20260925 15:12:19.789749 26675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:19.790150 26685 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } }
I20260925 15:12:19.796461 26687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f43e28390a64442991c844cce0013b7. Latest consensus state: current_term: 1 leader_uuid: "7f43e28390a64442991c844cce0013b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } } }
I20260925 15:12:19.797627 26687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:19.797438 26686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f43e28390a64442991c844cce0013b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f43e28390a64442991c844cce0013b7" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 43005 } } }
I20260925 15:12:19.797917 26686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:19.801877 26678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:19.782666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36081 (local address 127.4.205.254:43005)
0925 15:12:19.782890 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:19.782897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:19.783230 (+ 333us) server_negotiation.cc:407] Connection header received
0925 15:12:19.783401 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:19.783404 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:19.783492 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:19.783587 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:19.785405 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:19.786216 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:19.787988 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:19.788960 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:19.789140 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:19.792457 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:19.792466 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:19.792474 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:19.792523 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:19.797039 (+ 4516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:19.798078 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:19.798086 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:19.798092 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:19.798327 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:19.798980 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:19.798985 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:19.798989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:19.799428 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:19.799782 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:19.801091 (+ 1309us) server_negotiation.cc:299] Negotiation successful
0925 15:12:19.801340 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 15:12:19.802695 26694 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:19.808254 26694 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:19.818188 26694 catalog_manager.cc:1347] Generated new cluster ID: 0c7cd01965ec4da68ecb49779a6c50d4
I20260925 15:12:19.818287 26694 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:19.830120 26694 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:19.831338 26694 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:19.836761 26694 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f43e28390a64442991c844cce0013b7: Generated new TSK 0
I20260925 15:12:19.837616 26694 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:19.843214 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26608
I20260925 15:12:19.844844 26670 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:19.929832 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26608
W20260925 15:12:19.946022 26593 connection.cc:535] client connection to 127.4.205.254:43005 recv error: Network error: failed to read from TLS socket (remote: 127.4.205.254:43005): Connection reset by peer (error 104)
2026-09-25T15:12:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1895 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:20 dist-test-slave-64fx krb5kdc[26707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:20 dist-test-slave-64fx krb5kdc[26707](info): set up 2 sockets
Sep 25 15:12:20 dist-test-slave-64fx krb5kdc[26707](info): commencing operation
krb5kdc: starting...
W20260925 15:12:20.998062 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:21 dist-test-slave-64fx krb5kdc[26707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:21.224908 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33279
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:33279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:21.529163 26720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:21.529428 26720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:21.529552 26720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:21.536109 26720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:21.536206 26720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:21.536245 26720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:21.536322 26720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:21.536381 26720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:21.545262 26720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:33279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:33279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:21.546366 26720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:21.547520 26720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:21.558287 26726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:21.558249 26728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:21.558249 26725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:21.559235 26720 server_base.cc:1034] running on GCE node
I20260925 15:12:21.559892 26720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:21.561210 26720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:21.562428 26720 hybrid_clock.cc:648] HybridClock initialized: now 1790349141562394 us; error 55 us; skew 500 ppm
I20260925 15:12:21.562744 26720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:21 dist-test-slave-64fx krb5kdc[26707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349141, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:21.565891 26720 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:21.567075 26720 webserver.cc:460] Webserver started at http://127.4.205.254:46029/ using document root <none> and password file <none>
I20260925 15:12:21.567644 26720 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:21.567739 26720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:21.567962 26720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:21.570621 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "462a025ec48644e38a62fd769dfb5431"
format_stamp: "Formatted at 2026-09-25 15:12:21 on dist-test-slave-64fx"
server_key: "9b32cc9a4cf99a85eb9becfe1814c32c"
server_key_iv: "afdf9361dc3d2bce48ccc9e2e26ddaf2"
server_key_version: "encryptionkey@0"
I20260925 15:12:21.571316 26720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "462a025ec48644e38a62fd769dfb5431"
format_stamp: "Formatted at 2026-09-25 15:12:21 on dist-test-slave-64fx"
server_key: "9b32cc9a4cf99a85eb9becfe1814c32c"
server_key_iv: "afdf9361dc3d2bce48ccc9e2e26ddaf2"
server_key_version: "encryptionkey@0"
I20260925 15:12:21.576020 26720 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 15:12:21.579020 26735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:21.580155 26720 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:12:21.580273 26720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "462a025ec48644e38a62fd769dfb5431"
format_stamp: "Formatted at 2026-09-25 15:12:21 on dist-test-slave-64fx"
server_key: "9b32cc9a4cf99a85eb9becfe1814c32c"
server_key_iv: "afdf9361dc3d2bce48ccc9e2e26ddaf2"
server_key_version: "encryptionkey@0"
I20260925 15:12:21.580410 26720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:21.593708 26720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:21.597323 26720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:21.597523 26720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:21.614820 26720 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:33279
I20260925 15:12:21.614848 26786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:33279 every 8 connection(s)
I20260925 15:12:21.616094 26720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:21.619436 26787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:21.625448 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26720
I20260925 15:12:21.625649 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:21.626076 4919 external_mini_cluster.cc:1428] Setting key b118e6b066d3b0afc1b1c6d4323ee906
I20260925 15:12:21.629122 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431: Bootstrap starting.
I20260925 15:12:21.632030 26787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:21.633170 26787 log.cc:824] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:21.635931 26787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431: No bootstrap required, opened a new log
I20260925 15:12:21.638422 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:21.627626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59205 (local address 127.4.205.254:33279)
0925 15:12:21.628329 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:21.628339 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:21.628376 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:12:21.630051 (+ 1675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:21.630071 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:21.630427 (+ 356us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:21.630713 (+ 286us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:21.631226 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:21.632453 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:21.634208 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:21.634863 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:21.634994 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:21.636206 (+ 1212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:21.636212 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:21.636223 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:21.636717 (+ 494us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:21.636788 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:21.636810 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:21.636930 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:21.637231 (+ 301us) server_negotiation.cc:299] Negotiation successful
0925 15:12:21.637425 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":324,"threads_started":1}
I20260925 15:12:21.641992 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:21.644155 26787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } }
I20260925 15:12:21.644563 26787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:21.644728 26787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 462a025ec48644e38a62fd769dfb5431, State: Initialized, Role: FOLLOWER
I20260925 15:12:21.645349 26787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } }
I20260925 15:12:21.645674 26787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:21.645812 26787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:21.645999 26787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:21.648767 26787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } }
I20260925 15:12:21.649192 26787 leader_election.cc:302] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 462a025ec48644e38a62fd769dfb5431; no voters:
I20260925 15:12:21.649829 26787 leader_election.cc:288] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:21.649892 26797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:21.651243 26797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [term 1 LEADER]: Becoming Leader. State: Replica: 462a025ec48644e38a62fd769dfb5431, State: Running, Role: LEADER
I20260925 15:12:21.651691 26787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:21.651849 26797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } }
I20260925 15:12:21.654404 26790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:21.647024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43249 (local address 127.4.205.254:33279)
0925 15:12:21.647188 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:21.647193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:21.647668 (+ 475us) server_negotiation.cc:407] Connection header received
0925 15:12:21.647811 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:21.647814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:21.647865 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:21.647928 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:21.648403 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:21.649201 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:21.651053 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:21.651750 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:21.651874 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:21.653044 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:21.653050 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:21.653054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:21.653072 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:21.653151 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:21.653178 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:21.653327 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:21.653873 (+ 546us) server_negotiation.cc:299] Negotiation successful
0925 15:12:21.653940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:21.658329 26798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "462a025ec48644e38a62fd769dfb5431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } } }
I20260925 15:12:21.658686 26798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:21.658941 26799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 462a025ec48644e38a62fd769dfb5431. Latest consensus state: current_term: 1 leader_uuid: "462a025ec48644e38a62fd769dfb5431" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "462a025ec48644e38a62fd769dfb5431" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 33279 } } }
I20260925 15:12:21.659245 26799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:21.660982 26807 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:21.665825 26807 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:21.674074 26807 catalog_manager.cc:1347] Generated new cluster ID: c9ea6492c214439a93de8c2e6f71fc9b
I20260925 15:12:21.674176 26807 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:21.695876 26807 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:21.697001 26807 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:21.705646 26807 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 462a025ec48644e38a62fd769dfb5431: Generated new TSK 0
I20260925 15:12:21.706374 26807 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:21.723532 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26720
W20260925 15:12:21.724596 26782 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59205
W20260925 15:12:21.725286 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26720: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:21.725440 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26720
W20260925 15:12:21.741631 26705 connection.cc:535] client connection to 127.4.205.254:33279 recv error: Network error: failed to read from TLS socket (remote: 127.4.205.254:33279): Connection reset by peer (error 104)
2026-09-25T15:12:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1795 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:21 dist-test-slave-64fx krb5kdc[26818](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:21 dist-test-slave-64fx krb5kdc[26818](info): set up 2 sockets
Sep 25 15:12:21 dist-test-slave-64fx krb5kdc[26818](info): commencing operation
krb5kdc: starting...
W20260925 15:12:22.785980 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:22 dist-test-slave-64fx krb5kdc[26818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:22.998281 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39165
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:35289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:23.310104 26831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:23.310375 26831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:23.310460 26831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:23.317395 26831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:23.317489 26831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:23.317531 26831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:23.317668 26831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:23.317729 26831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:23.327193 26831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:35289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39165
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:23.328297 26831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:23.329449 26831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:23.340994 26839 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:23.341079 26836 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:23.341058 26837 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:23.341910 26831 server_base.cc:1034] running on GCE node
I20260925 15:12:23.342669 26831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:23.343956 26831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:23.345191 26831 hybrid_clock.cc:648] HybridClock initialized: now 1790349143345151 us; error 53 us; skew 500 ppm
I20260925 15:12:23.345525 26831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:23 dist-test-slave-64fx krb5kdc[26818](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349143, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:23.354871 26831 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:23.356251 26831 webserver.cc:460] Webserver started at http://127.4.205.254:42763/ using document root <none> and password file <none>
I20260925 15:12:23.356894 26831 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:23.357029 26831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:23.357316 26831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:23.360153 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "d57588e4ca924310b2378fa16ec7367d"
format_stamp: "Formatted at 2026-09-25 15:12:23 on dist-test-slave-64fx"
server_key: "63221b351a22a1453b9ddac619354c56"
server_key_iv: "4e936e76d12db66c51ab01dd8aaf8c00"
server_key_version: "encryptionkey@0"
I20260925 15:12:23.360917 26831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "d57588e4ca924310b2378fa16ec7367d"
format_stamp: "Formatted at 2026-09-25 15:12:23 on dist-test-slave-64fx"
server_key: "63221b351a22a1453b9ddac619354c56"
server_key_iv: "4e936e76d12db66c51ab01dd8aaf8c00"
server_key_version: "encryptionkey@0"
I20260925 15:12:23.365716 26831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.008s sys 0.000s
I20260925 15:12:23.368892 26846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:23.370190 26831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 15:12:23.370357 26831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "d57588e4ca924310b2378fa16ec7367d"
format_stamp: "Formatted at 2026-09-25 15:12:23 on dist-test-slave-64fx"
server_key: "63221b351a22a1453b9ddac619354c56"
server_key_iv: "4e936e76d12db66c51ab01dd8aaf8c00"
server_key_version: "encryptionkey@0"
I20260925 15:12:23.370535 26831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:23.375970 26831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:23.379581 26831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:23.379856 26831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:23.397321 26831 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:39165
I20260925 15:12:23.397362 26897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:39165 every 8 connection(s)
I20260925 15:12:23.398617 26831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:23.401853 26898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:23.407480 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26831
I20260925 15:12:23.407676 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:23.408102 4919 external_mini_cluster.cc:1428] Setting key 4908311f30088b6f11b7f0ec331f667c
I20260925 15:12:23.411715 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d: Bootstrap starting.
I20260925 15:12:23.414705 26898 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:23.415234 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:23.409811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55795 (local address 127.4.205.254:39165)
0925 15:12:23.410516 (+ 705us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:23.410525 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:23.410559 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:12:23.412046 (+ 1487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:23.412066 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:23.412374 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:23.412717 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:23.413708 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:23.413713 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:23.413725 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:23.413737 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:23.413789 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:23.413805 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:23.414013 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:23.414175 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:12:23.414334 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":303,"threads_started":1}
I20260925 15:12:23.415872 26898 log.cc:824] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:23.418725 26898 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d: No bootstrap required, opened a new log
I20260925 15:12:23.418826 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:23.425328 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:23.423003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35485 (local address 127.4.205.254:39165)
0925 15:12:23.423234 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:23.423244 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:23.423496 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:12:23.423698 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:23.423704 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:23.423757 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:23.423862 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:23.424689 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:23.424695 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:23.424700 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:23.424718 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:23.424761 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:23.424774 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:23.424913 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:23.425000 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:12:23.425053 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:12:23.426259 26898 raft_consensus.cc:348] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } }
I20260925 15:12:23.426659 26898 raft_consensus.cc:374] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:23.426767 26898 raft_consensus.cc:729] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d57588e4ca924310b2378fa16ec7367d, State: Initialized, Role: FOLLOWER
I20260925 15:12:23.427242 26898 consensus_queue.cc:260] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } }
I20260925 15:12:23.427448 26898 raft_consensus.cc:388] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:23.427583 26898 raft_consensus.cc:482] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:23.427736 26898 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:23.429723 26898 raft_consensus.cc:504] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } }
I20260925 15:12:23.430181 26898 leader_election.cc:302] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d57588e4ca924310b2378fa16ec7367d; no voters:
I20260925 15:12:23.430738 26898 leader_election.cc:288] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:23.430835 26908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:23.431519 26908 raft_consensus.cc:686] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [term 1 LEADER]: Becoming Leader. State: Replica: d57588e4ca924310b2378fa16ec7367d, State: Running, Role: LEADER
I20260925 15:12:23.432118 26908 consensus_queue.cc:237] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } }
I20260925 15:12:23.432651 26898 sys_catalog.cc:564] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:23.434643 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d57588e4ca924310b2378fa16ec7367d. Latest consensus state: current_term: 1 leader_uuid: "d57588e4ca924310b2378fa16ec7367d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } } }
I20260925 15:12:23.435004 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:23.435540 26910 sys_catalog.cc:455] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d57588e4ca924310b2378fa16ec7367d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d57588e4ca924310b2378fa16ec7367d" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39165 } } }
I20260925 15:12:23.435909 26910 sys_catalog.cc:458] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:23.440492 26915 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:23.445581 26915 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:23.453100 26915 catalog_manager.cc:1347] Generated new cluster ID: d18090053d6043bf8db5d60e86176d8a
I20260925 15:12:23.453199 26915 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:23.467600 26915 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:23.467963 26915 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:23.474561 26915 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d57588e4ca924310b2378fa16ec7367d: Generated new TSK 0
I20260925 15:12:23.475289 26915 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:23.481695 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26831
W20260925 15:12:23.482729 26893 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55795
W20260925 15:12:23.483357 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26831: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:23.483495 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26831
2026-09-25T15:12:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1755 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:23 dist-test-slave-64fx krb5kdc[26930](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:23 dist-test-slave-64fx krb5kdc[26930](info): set up 2 sockets
Sep 25 15:12:23 dist-test-slave-64fx krb5kdc[26930](info): commencing operation
krb5kdc: starting...
W20260925 15:12:24.562031 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:24 dist-test-slave-64fx krb5kdc[26930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:24.778555 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35365
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:38301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:35365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:25.092057 26943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:25.092343 26943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:25.092453 26943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:25.099646 26943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:25.099757 26943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:25.099803 26943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:25.099864 26943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:25.099908 26943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:25.109879 26943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:38301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:35365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:35365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:25.111107 26943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:25.112402 26943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:25.123847 26948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:25.124095 26949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:25.124130 26951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:25.124676 26943 server_base.cc:1034] running on GCE node
I20260925 15:12:25.125429 26943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:25.126868 26943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:25.128059 26943 hybrid_clock.cc:648] HybridClock initialized: now 1790349145128024 us; error 24 us; skew 500 ppm
I20260925 15:12:25.128407 26943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:25 dist-test-slave-64fx krb5kdc[26930](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349145, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:25.131587 26943 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:25.133004 26943 webserver.cc:460] Webserver started at http://127.4.205.254:42601/ using document root <none> and password file <none>
I20260925 15:12:25.133741 26943 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:25.133876 26943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:25.134192 26943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:25.137023 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "0056ae39e3cb4c788e52506a410cd3af"
format_stamp: "Formatted at 2026-09-25 15:12:25 on dist-test-slave-64fx"
server_key: "fa2efe84a2450f8e86757876e086eb8a"
server_key_iv: "df3bd35199c319df2500884853cb6d51"
server_key_version: "encryptionkey@0"
I20260925 15:12:25.137866 26943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "0056ae39e3cb4c788e52506a410cd3af"
format_stamp: "Formatted at 2026-09-25 15:12:25 on dist-test-slave-64fx"
server_key: "fa2efe84a2450f8e86757876e086eb8a"
server_key_iv: "df3bd35199c319df2500884853cb6d51"
server_key_version: "encryptionkey@0"
I20260925 15:12:25.142779 26943 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 15:12:25.146035 26958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:25.147199 26943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260925 15:12:25.147372 26943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "0056ae39e3cb4c788e52506a410cd3af"
format_stamp: "Formatted at 2026-09-25 15:12:25 on dist-test-slave-64fx"
server_key: "fa2efe84a2450f8e86757876e086eb8a"
server_key_iv: "df3bd35199c319df2500884853cb6d51"
server_key_version: "encryptionkey@0"
I20260925 15:12:25.147555 26943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:25.163341 26943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:25.167222 26943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:25.167492 26943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:25.185128 26943 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:35365
I20260925 15:12:25.185146 27009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:35365 every 8 connection(s)
I20260925 15:12:25.186617 26943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:25.188061 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 26943
I20260925 15:12:25.188247 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:25.188745 4919 external_mini_cluster.cc:1428] Setting key d004d4ae886f25a4ac5f525ccaacc1a0
I20260925 15:12:25.191025 27010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 15:12:25 dist-test-slave-64fx krb5kdc[26930](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:25.203482 27010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af: Bootstrap starting.
I20260925 15:12:25.206854 27010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:25.208032 27010 log.cc:824] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:25.210906 27010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af: No bootstrap required, opened a new log
I20260925 15:12:25.211673 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:25.190522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42443 (local address 127.4.205.254:35365)
0925 15:12:25.191234 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:25.191246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:25.191303 (+ 57us) server_negotiation.cc:407] Connection header received
0925 15:12:25.193428 (+ 2125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:25.193453 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:25.193977 (+ 524us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:25.194409 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:25.195612 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:25.197580 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:25.198811 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:25.199770 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:25.199951 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:25.204252 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:25.204260 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:25.204273 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:25.204310 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:25.207419 (+ 3109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:25.208281 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:25.208288 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:25.208294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:25.208448 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:25.208854 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:25.208864 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:25.208868 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:25.209380 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:25.209712 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:25.210181 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 15:12:25.210583 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":281,"threads_started":1}
I20260925 15:12:25.215354 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:25.218837 27010 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } }
I20260925 15:12:25.219194 27010 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:25.219304 27010 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0056ae39e3cb4c788e52506a410cd3af, State: Initialized, Role: FOLLOWER
I20260925 15:12:25.220077 27010 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } }
I20260925 15:12:25.220333 27010 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:25.220497 27010 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:25.220695 27010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:25.223399 27010 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } }
I20260925 15:12:25.223877 27010 leader_election.cc:302] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0056ae39e3cb4c788e52506a410cd3af; no voters:
I20260925 15:12:25.224509 27010 leader_election.cc:288] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:25.224617 27020 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:25.226375 27020 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [term 1 LEADER]: Becoming Leader. State: Replica: 0056ae39e3cb4c788e52506a410cd3af, State: Running, Role: LEADER
I20260925 15:12:25.227010 27010 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:25.226887 27020 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } }
I20260925 15:12:25.230036 27022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0056ae39e3cb4c788e52506a410cd3af. Latest consensus state: current_term: 1 leader_uuid: "0056ae39e3cb4c788e52506a410cd3af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } } }
I20260925 15:12:25.230394 27022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:25.231005 27021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0056ae39e3cb4c788e52506a410cd3af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0056ae39e3cb4c788e52506a410cd3af" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 35365 } } }
I20260925 15:12:25.231355 27021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:25.234155 27027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:25.238845 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:25.219502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34629 (local address 127.4.205.254:35365)
0925 15:12:25.219684 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:25.219690 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:25.220198 (+ 508us) server_negotiation.cc:407] Connection header received
0925 15:12:25.220417 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:25.220421 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:25.220513 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:25.220621 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:25.222116 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:25.223212 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:25.224544 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:25.225387 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:25.225506 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:25.228139 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:25.228145 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:25.228150 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:25.228184 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:25.234469 (+ 6285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:25.235481 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:25.235488 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:25.235491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:25.235694 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:25.236187 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:25.236194 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:25.236200 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:25.236643 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:25.238069 (+ 1426us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:25.238215 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:12:25.238460 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:12:25.240092 27027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:25.250038 27027 catalog_manager.cc:1347] Generated new cluster ID: 76147ac0568646079ae56061629232bc
I20260925 15:12:25.250145 27027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:25.268008 27027 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:25.269706 27027 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:25.283120 27027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0056ae39e3cb4c788e52506a410cd3af: Generated new TSK 0
I20260925 15:12:25.284152 27027 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:25.291368 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 26943
I20260925 15:12:25.292742 27005 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:25.382470 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 26943
2026-09-25T15:12:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1901 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:25 dist-test-slave-64fx krb5kdc[27042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:25 dist-test-slave-64fx krb5kdc[27042](info): set up 2 sockets
Sep 25 15:12:25 dist-test-slave-64fx krb5kdc[27042](info): commencing operation
krb5kdc: starting...
W20260925 15:12:26.450067 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:26 dist-test-slave-64fx krb5kdc[27042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:26.674635 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:38439
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:26.979794 27055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:26.980037 27055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:26.980116 27055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:26.986867 27055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:26.986961 27055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:26.987095 27055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:26.987195 27055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:26.987277 27055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:26.996328 27055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:26.997509 27055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:26.998749 27055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:27.010614 27063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:27.010668 27060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:27.010583 27061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:27.011655 27055 server_base.cc:1034] running on GCE node
I20260925 15:12:27.012353 27055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:27.013707 27055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:27.014933 27055 hybrid_clock.cc:648] HybridClock initialized: now 1790349147014898 us; error 44 us; skew 500 ppm
I20260925 15:12:27.015280 27055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:27 dist-test-slave-64fx krb5kdc[27042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349147, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:27.018216 27055 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:27.019635 27055 webserver.cc:460] Webserver started at http://127.4.205.254:42659/ using document root <none> and password file <none>
I20260925 15:12:27.020290 27055 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:27.020429 27055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:27.020717 27055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:27.023602 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "f899411ab4b740d5b6cf937de2cde2b3"
format_stamp: "Formatted at 2026-09-25 15:12:27 on dist-test-slave-64fx"
server_key: "dcd790abdcb865945529b593369d7ca8"
server_key_iv: "19a72825214b01cabc413f4784f63f1a"
server_key_version: "encryptionkey@0"
I20260925 15:12:27.024354 27055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "f899411ab4b740d5b6cf937de2cde2b3"
format_stamp: "Formatted at 2026-09-25 15:12:27 on dist-test-slave-64fx"
server_key: "dcd790abdcb865945529b593369d7ca8"
server_key_iv: "19a72825214b01cabc413f4784f63f1a"
server_key_version: "encryptionkey@0"
I20260925 15:12:27.029129 27055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.006s
I20260925 15:12:27.032465 27070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:27.033615 27055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260925 15:12:27.033774 27055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "f899411ab4b740d5b6cf937de2cde2b3"
format_stamp: "Formatted at 2026-09-25 15:12:27 on dist-test-slave-64fx"
server_key: "dcd790abdcb865945529b593369d7ca8"
server_key_iv: "19a72825214b01cabc413f4784f63f1a"
server_key_version: "encryptionkey@0"
I20260925 15:12:27.033949 27055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:27.049423 27055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:27.052881 27055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:27.053153 27055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:27.070359 27055 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:38439
I20260925 15:12:27.070385 27121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:38439 every 8 connection(s)
I20260925 15:12:27.071481 27055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:27.074663 27122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:27.076220 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27055
I20260925 15:12:27.076385 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:27.076857 4919 external_mini_cluster.cc:1428] Setting key f6fdba81f6924fbe7f039fb91cb75682
I20260925 15:12:27.086449 27122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3: Bootstrap starting.
I20260925 15:12:27.088455 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:27.078550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38887 (local address 127.4.205.254:38439)
0925 15:12:27.079258 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:27.079268 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:27.079307 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:12:27.080768 (+ 1461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:27.080791 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:27.081180 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:27.081551 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:27.082062 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:27.083229 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:27.084496 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:27.085137 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:27.085332 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:27.086439 (+ 1107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:27.086446 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:27.086456 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:27.087003 (+ 547us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:27.087072 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:27.087099 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:27.087221 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:27.087437 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:12:27.087670 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":320,"threads_started":1}
I20260925 15:12:27.089263 27122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:27.090337 27122 log.cc:824] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:27.091885 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:27.093122 27122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3: No bootstrap required, opened a new log
I20260925 15:12:27.102588 27125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:27.096503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39925 (local address 127.4.205.254:38439)
0925 15:12:27.096689 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:27.096699 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:27.097082 (+ 383us) server_negotiation.cc:407] Connection header received
0925 15:12:27.097208 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:27.097213 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:27.097282 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:27.097387 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:27.097910 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:27.098977 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:27.100129 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:27.100814 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:27.100973 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:27.101906 (+ 933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:27.101912 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:27.101916 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:27.101935 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:27.101991 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:27.102010 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:27.102113 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:27.102205 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 15:12:27.102284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:27.102550 27122 raft_consensus.cc:348] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } }
I20260925 15:12:27.102964 27122 raft_consensus.cc:374] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:27.103077 27122 raft_consensus.cc:729] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f899411ab4b740d5b6cf937de2cde2b3, State: Initialized, Role: FOLLOWER
I20260925 15:12:27.103490 27122 consensus_queue.cc:260] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } }
I20260925 15:12:27.103716 27122 raft_consensus.cc:388] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:27.103833 27122 raft_consensus.cc:482] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:27.103986 27122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:27.106014 27122 raft_consensus.cc:504] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } }
I20260925 15:12:27.106424 27122 leader_election.cc:302] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f899411ab4b740d5b6cf937de2cde2b3; no voters:
I20260925 15:12:27.106853 27122 leader_election.cc:288] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:27.106977 27132 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:27.108059 27132 raft_consensus.cc:686] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [term 1 LEADER]: Becoming Leader. State: Replica: f899411ab4b740d5b6cf937de2cde2b3, State: Running, Role: LEADER
I20260925 15:12:27.108654 27132 consensus_queue.cc:237] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } }
I20260925 15:12:27.108762 27122 sys_catalog.cc:564] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:27.111353 27133 sys_catalog.cc:455] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f899411ab4b740d5b6cf937de2cde2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } } }
I20260925 15:12:27.111776 27133 sys_catalog.cc:458] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:27.112258 27134 sys_catalog.cc:455] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f899411ab4b740d5b6cf937de2cde2b3. Latest consensus state: current_term: 1 leader_uuid: "f899411ab4b740d5b6cf937de2cde2b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f899411ab4b740d5b6cf937de2cde2b3" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 38439 } } }
I20260925 15:12:27.112615 27134 sys_catalog.cc:458] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:27.113996 27138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:27.118410 27138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:27.127867 27138 catalog_manager.cc:1347] Generated new cluster ID: 1f495e927d7b4cb0bdbd0d2f8d208ff0
I20260925 15:12:27.127964 27138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:27.141070 27138 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:27.142228 27138 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:27.158751 27138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f899411ab4b740d5b6cf937de2cde2b3: Generated new TSK 0
I20260925 15:12:27.159483 27138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:27.166086 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27055
W20260925 15:12:27.167078 27117 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38887
W20260925 15:12:27.167733 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27055: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:27.167865 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27055
2026-09-25T15:12:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1784 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:27 dist-test-slave-64fx krb5kdc[27153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:27 dist-test-slave-64fx krb5kdc[27153](info): set up 2 sockets
Sep 25 15:12:27 dist-test-slave-64fx krb5kdc[27153](info): commencing operation
krb5kdc: starting...
W20260925 15:12:28.226074 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:28 dist-test-slave-64fx krb5kdc[27153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349148, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:28.463938 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40127
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:37221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:40127
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:28.777657 27166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:28.777995 27166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:28.778162 27166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:28.785053 27166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:28.785197 27166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:28.785305 27166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:28.785413 27166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:28.785496 27166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:28.794799 27166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:37221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:40127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:40127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:28.796020 27166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:28.797243 27166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:28.808706 27174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:28.808733 27172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:28.808775 27171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:28.809461 27166 server_base.cc:1034] running on GCE node
I20260925 15:12:28.810168 27166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:28.811360 27166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:28.812541 27166 hybrid_clock.cc:648] HybridClock initialized: now 1790349148812485 us; error 51 us; skew 500 ppm
I20260925 15:12:28.812860 27166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:28 dist-test-slave-64fx krb5kdc[27153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349148, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:28.821151 27166 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:28.822517 27166 webserver.cc:460] Webserver started at http://127.4.205.254:40955/ using document root <none> and password file <none>
I20260925 15:12:28.823112 27166 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:28.823235 27166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:28.823518 27166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:28.826363 27166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "0ddca21b6bb64cb6a24b138f714e19f6"
format_stamp: "Formatted at 2026-09-25 15:12:28 on dist-test-slave-64fx"
server_key: "a513cd926a79f865d2b92efb87f2f714"
server_key_iv: "4cd01a4b633654301392a73bda294e7e"
server_key_version: "encryptionkey@0"
I20260925 15:12:28.827066 27166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "0ddca21b6bb64cb6a24b138f714e19f6"
format_stamp: "Formatted at 2026-09-25 15:12:28 on dist-test-slave-64fx"
server_key: "a513cd926a79f865d2b92efb87f2f714"
server_key_iv: "4cd01a4b633654301392a73bda294e7e"
server_key_version: "encryptionkey@0"
I20260925 15:12:28.832029 27166 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 15:12:28.835172 27181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:28.836364 27166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:12:28.836498 27166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "0ddca21b6bb64cb6a24b138f714e19f6"
format_stamp: "Formatted at 2026-09-25 15:12:28 on dist-test-slave-64fx"
server_key: "a513cd926a79f865d2b92efb87f2f714"
server_key_iv: "4cd01a4b633654301392a73bda294e7e"
server_key_version: "encryptionkey@0"
I20260925 15:12:28.836678 27166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:28.841923 27166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:28.845460 27166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:28.845758 27166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:28.863210 27166 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:40127
I20260925 15:12:28.863250 27232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:40127 every 8 connection(s)
I20260925 15:12:28.864506 27166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:28.865147 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27166
I20260925 15:12:28.865346 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:28.865878 4919 external_mini_cluster.cc:1428] Setting key 8f39e7b84053d24ff89304d1add8dd3e
I20260925 15:12:28.868854 27233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:28.874089 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:28.867484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47151 (local address 127.4.205.254:40127)
0925 15:12:28.868255 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:28.868269 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:28.868314 (+ 45us) server_negotiation.cc:407] Connection header received
0925 15:12:28.870565 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:28.870591 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:28.870968 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:28.871361 (+ 393us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:28.872279 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:28.872287 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:28.872304 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:28.872325 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:28.872414 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:28.872439 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:28.872643 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:28.872768 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:12:28.872939 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":312,"threads_started":1}
I20260925 15:12:28.877314 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:28.881754 27233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6: Bootstrap starting.
I20260925 15:12:28.884670 27236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:28.882411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49927 (local address 127.4.205.254:40127)
0925 15:12:28.882582 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:28.882587 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:28.882943 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:12:28.883050 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:28.883055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:28.883111 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:28.883234 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:28.884050 (+ 816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:28.884055 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:28.884061 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:12:28.884077 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:28.884123 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:12:28.884145 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:28.884257 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:28.884324 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:12:28.884379 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:12:28.885327 27233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:28.886438 27233 log.cc:824] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:28.888814 27233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6: No bootstrap required, opened a new log
I20260925 15:12:28.895664 27233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } }
I20260925 15:12:28.896013 27233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:12:28.896111 27233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ddca21b6bb64cb6a24b138f714e19f6, State: Initialized, Role: FOLLOWER
I20260925 15:12:28.896600 27233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } }
I20260925 15:12:28.896818 27233 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:12:28.896977 27233 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:12:28.897151 27233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:28.899257 27233 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } }
I20260925 15:12:28.899719 27233 leader_election.cc:302] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ddca21b6bb64cb6a24b138f714e19f6; no voters:
I20260925 15:12:28.900272 27233 leader_election.cc:288] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:12:28.900429 27243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:28.901103 27243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [term 1 LEADER]: Becoming Leader. State: Replica: 0ddca21b6bb64cb6a24b138f714e19f6, State: Running, Role: LEADER
I20260925 15:12:28.901683 27243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } }
I20260925 15:12:28.902283 27233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:28.904280 27245 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ddca21b6bb64cb6a24b138f714e19f6. Latest consensus state: current_term: 1 leader_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } } }
I20260925 15:12:28.904692 27245 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:28.905509 27244 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ddca21b6bb64cb6a24b138f714e19f6" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 40127 } } }
I20260925 15:12:28.905958 27244 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:28.907958 27250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:28.912910 27250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:28.920995 27250 catalog_manager.cc:1347] Generated new cluster ID: 9dd37d8f829145aab61df9e5e46f06f9
I20260925 15:12:28.921097 27250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:28.940281 27250 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:12:28.940778 27250 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:28.947489 27250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ddca21b6bb64cb6a24b138f714e19f6: Generated new TSK 0
I20260925 15:12:28.948349 27250 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:28.960649 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27166
W20260925 15:12:28.961733 27228 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47151
W20260925 15:12:28.962380 4919 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27166: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 15:12:28.962518 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27166
2026-09-25T15:12:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1794 ms)
[----------] 18 tests from AuthTokenIssuingTest (32877 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:29 dist-test-slave-64fx krb5kdc[27264](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:29 dist-test-slave-64fx krb5kdc[27264](info): set up 2 sockets
Sep 25 15:12:29 dist-test-slave-64fx krb5kdc[27264](info): commencing operation
krb5kdc: starting...
W20260925 15:12:30.046095 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.006s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:30 dist-test-slave-64fx krb5kdc[27264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:30.280320 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39097
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:30.597671 27277 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:30.597951 27277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:30.598014 27277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:30.598083 27277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:30.604766 27277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:30.604844 27277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:30.604900 27277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:30.604949 27277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:30.605003 27277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:30.614053 27277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:39097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:30.615191 27277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:30.616366 27277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:30.627525 27282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:30.627655 27285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:30.628130 27283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:30.628434 27277 server_base.cc:1034] running on GCE node
I20260925 15:12:30.629148 27277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:30.630515 27277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:30.631721 27277 hybrid_clock.cc:648] HybridClock initialized: now 1790349150631650 us; error 47 us; skew 500 ppm
I20260925 15:12:30.632035 27277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:30 dist-test-slave-64fx krb5kdc[27264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:30.635088 27277 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:30.636397 27277 webserver.cc:460] Webserver started at http://127.4.205.254:44469/ using document root <none> and password file <none>
I20260925 15:12:30.637076 27277 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:30.637205 27277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:30.637511 27277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:30.640234 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "2d52a2cee8324b268190f8cd79419995"
format_stamp: "Formatted at 2026-09-25 15:12:30 on dist-test-slave-64fx"
server_key: "fe189cfd80e76cd8d1869cec5e51b646"
server_key_iv: "5188d02872039800900f8c88f9a7dc69"
server_key_version: "encryptionkey@0"
I20260925 15:12:30.641016 27277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "2d52a2cee8324b268190f8cd79419995"
format_stamp: "Formatted at 2026-09-25 15:12:30 on dist-test-slave-64fx"
server_key: "fe189cfd80e76cd8d1869cec5e51b646"
server_key_iv: "5188d02872039800900f8c88f9a7dc69"
server_key_version: "encryptionkey@0"
I20260925 15:12:30.646314 27277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 15:12:30.649354 27292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:30.650597 27277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:12:30.650763 27277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "2d52a2cee8324b268190f8cd79419995"
format_stamp: "Formatted at 2026-09-25 15:12:30 on dist-test-slave-64fx"
server_key: "fe189cfd80e76cd8d1869cec5e51b646"
server_key_iv: "5188d02872039800900f8c88f9a7dc69"
server_key_version: "encryptionkey@0"
I20260925 15:12:30.650969 27277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:30.666929 27277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:30.671234 27277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:30.671516 27277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:30.689301 27277 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:39097
I20260925 15:12:30.689322 27343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:39097 every 8 connection(s)
I20260925 15:12:30.690613 27277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:30.693447 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27277
I20260925 15:12:30.693711 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:30.694475 4919 external_mini_cluster.cc:1428] Setting key d432b6d7aacd46f2fbacb6c6747b9c6c
I20260925 15:12:30.694499 27344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:30.704034 27344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:30 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.253@KRBTEST.COM" created.
I20260925 15:12:30.727316 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.707506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39097 (local address 127.0.0.1:37152)
0925 15:12:30.708576 (+ 1070us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.708609 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:30.710320 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:30.711116 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:30.711123 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:30.711808 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:30.712121 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:30.712138 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:30.713745 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:30.713754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:30.715704 (+ 1950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:30.715722 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:30.716644 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:30.716654 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:30.716851 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:30.717527 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:30.717580 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:30.720260 (+ 2680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:30.723501 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:30.723514 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:30.723528 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:30.723975 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:30.724427 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:30.724430 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:30.724432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:30.724572 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:30.725518 (+ 946us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:30.725523 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:30.725748 (+ 225us) client_negotiation.cc:769] Sending connection context
0925 15:12:30.725953 (+ 205us) client_negotiation.cc:240] Negotiation successful
0925 15:12:30.726333 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"thread_start_us":653,"threads_started":1}
I20260925 15:12:30.727341 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.707495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.4.205.254:39097)
0925 15:12:30.708271 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:30.708290 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:30.708736 (+ 446us) server_negotiation.cc:407] Connection header received
0925 15:12:30.710661 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:30.710667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:30.710902 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:30.711117 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:30.712339 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:30.713512 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:30.715856 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:30.716370 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:30.716582 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:30.720471 (+ 3889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:30.720479 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:30.720489 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:30.720516 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:30.723356 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:30.724111 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:30.724121 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:30.724127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:30.724314 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:30.724685 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:30.724692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:30.724698 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:30.725382 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:30.725609 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:30.726140 (+ 531us) server_negotiation.cc:299] Negotiation successful
0925 15:12:30.726524 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":544,"thread_start_us":340,"threads_started":1}
I20260925 15:12:30.730120 27344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:30.731320 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.730831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60474)
0925 15:12:30.731031 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.731166 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260925 15:12:30.731590 27294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
I20260925 15:12:30.733059 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.732685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60490)
0925 15:12:30.732851 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.732910 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260925 15:12:30.733244 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:45271: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
I20260925 15:12:30.752806 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:45271
--webserver_interface=127.4.205.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:12:30.782578 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } attempt: 1
I20260925 15:12:30.783797 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.783339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60504)
0925 15:12:30.783543 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.783662 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 15:12:30.784852 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.784515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60508)
0925 15:12:30.784643 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.784728 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:12:30.785094 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:45271: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
I20260925 15:12:30.853369 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } attempt: 2
I20260925 15:12:30.854617 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.854116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60516)
0925 15:12:30.854395 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.854480 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 15:12:30.855634 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.855276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60532)
0925 15:12:30.855422 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.855509 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 15:12:30.855832 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:45271: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
I20260925 15:12:30.947116 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } attempt: 3
I20260925 15:12:30.948217 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.947782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60542)
0925 15:12:30.947987 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.948087 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:12:30.949211 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:30.948865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60544)
0925 15:12:30.949010 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:30.949087 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:12:30.949448 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:45271: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
W20260925 15:12:31.061930 27351 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:31.062245 27351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:31.062355 27351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:31.062486 27351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:31.069135 27351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:31.069206 27351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:31.069258 27351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:31.069324 27351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:31.069411 27351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:31.078526 27351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:45271
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.205.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:31.079689 27351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:31.080932 27351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:31.091761 27356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:31.091733 27357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:31.091907 27359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:31.092420 27351 server_base.cc:1034] running on GCE node
I20260925 15:12:31.092660 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } attempt: 4
I20260925 15:12:31.093262 27351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:31.093823 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.093234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60552)
0925 15:12:31.093528 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.093655 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 15:12:31.094806 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.094472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60558)
0925 15:12:31.094594 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.094676 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 15:12:31.094978 27351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260925 15:12:31.095090 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:45271: Network error: Client connection negotiation failed: client connection to 127.4.205.253:45271: connect: Connection refused (error 111)
I20260925 15:12:31.096233 27351 hybrid_clock.cc:648] HybridClock initialized: now 1790349151096169 us; error 38 us; skew 500 ppm
I20260925 15:12:31.096522 27351 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:31.099596 27351 init.cc:377] Logged in from keytab as kudu/127.4.205.253@KRBTEST.COM (short username kudu)
I20260925 15:12:31.100944 27351 webserver.cc:460] Webserver started at http://127.4.205.253:44661/ using document root <none> and password file <none>
I20260925 15:12:31.101642 27351 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:31.101738 27351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:31.101990 27351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:31.104796 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data/instance:
uuid: "f802036a84614136bb1fddb5d210e372"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "264fad37cce604d4d3007bab18c88406"
server_key_iv: "1f7863281a21ab78d45fb4e201c3df5b"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.105509 27351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal/instance:
uuid: "f802036a84614136bb1fddb5d210e372"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "264fad37cce604d4d3007bab18c88406"
server_key_iv: "1f7863281a21ab78d45fb4e201c3df5b"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.110427 27351 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260925 15:12:31.113477 27366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:31.114771 27351 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:12:31.114948 27351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "f802036a84614136bb1fddb5d210e372"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "264fad37cce604d4d3007bab18c88406"
server_key_iv: "1f7863281a21ab78d45fb4e201c3df5b"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.115152 27351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:31.130475 27351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:31.134215 27351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:31.134485 27351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:31.152099 27351 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.253:45271
I20260925 15:12:31.152118 27417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.253:45271 every 8 connection(s)
I20260925 15:12:31.153370 27351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
I20260925 15:12:31.154874 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27351
I20260925 15:12:31.155072 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-1/wal/instance
I20260925 15:12:31.155583 4919 external_mini_cluster.cc:1428] Setting key 0c65871de6cc2efef92a518132e2ae2c
I20260925 15:12:31.157143 27418 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:31.163774 27418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.252@KRBTEST.COM" created.
I20260925 15:12:31.185133 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.167220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.4.205.254:39097)
0925 15:12:31.167432 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.167439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.168646 (+ 1207us) server_negotiation.cc:407] Connection header received
0925 15:12:31.170474 (+ 1828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.170483 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.170574 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.170696 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.172048 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.173151 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.174723 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.175486 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.175631 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.179005 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.179009 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.179012 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.179042 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.182034 (+ 2992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.182716 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.182724 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.182730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.182877 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.183241 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.183248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.183254 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.183663 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.183822 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.184450 (+ 628us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.184716 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 15:12:31.185645 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.167697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39097 (local address 127.0.0.1:37156)
0925 15:12:31.168455 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.168505 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.170212 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.170865 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.170872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.171352 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.171874 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.171891 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.173311 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.173316 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.174544 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.174561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.175681 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.175686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.175885 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.176581 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.176600 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.178788 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.182189 (+ 3401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.182194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.182204 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.182584 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.182986 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.182989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.182991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.183129 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.183781 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.183786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.184077 (+ 291us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.184276 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.184590 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":336,"threads_started":1}
I20260925 15:12:31.187392 27418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:31.200721 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.188125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60562)
0925 15:12:31.188276 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.188294 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.188375 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.189204 (+ 829us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.189208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.189688 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.189813 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.189821 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.191127 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.191135 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.191908 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.191917 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.192677 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.192682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.192833 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.193377 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.193392 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.195493 (+ 2101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.198391 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.198398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.198402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.198803 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.199133 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.199140 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.199144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.199270 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.199784 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.199790 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.199913 (+ 123us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.200067 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.200306 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:12:31.200809 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.188291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.4.205.253:45271)
0925 15:12:31.188860 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.188865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.188886 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:12:31.188921 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.188925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.189075 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.189211 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.189983 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.191004 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.192028 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.192541 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.192666 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.195682 (+ 3016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.195691 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.195702 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.195726 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.198280 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.198920 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.198935 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.198938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.199036 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.199372 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.199380 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.199384 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.199671 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.199870 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.200183 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.200419 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":434,"thread_start_us":329,"threads_started":1}
I20260925 15:12:31.202965 27418 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:31.203884 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.203481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36858)
0925 15:12:31.203652 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.203756 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:12:31.204124 27368 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.205586 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.205194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36866)
0925 15:12:31.205329 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.205436 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:12:31.205811 27418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
I20260925 15:12:31.209872 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:39429
--webserver_interface=127.4.205.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:12:31.255103 27418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 1
I20260925 15:12:31.256423 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.255803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36878)
0925 15:12:31.256064 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.256211 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 15:12:31.257647 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.257142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36882)
0925 15:12:31.257337 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.257454 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:12:31.257869 27418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.326100 27418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 2
I20260925 15:12:31.327236 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.326715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36896)
0925 15:12:31.326949 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.327056 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 15:12:31.328253 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.327881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36898)
0925 15:12:31.328019 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.328117 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:12:31.328480 27418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.394338 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } attempt: 5
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:31.408602 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.395187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60570 (local address 127.4.205.253:45271)
0925 15:12:31.395358 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.395362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.395385 (+ 23us) server_negotiation.cc:407] Connection header received
0925 15:12:31.395622 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.395626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.395686 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.395767 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.396727 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.397539 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.399239 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.399661 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.399749 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.403322 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.403326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.403329 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.403358 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.406017 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.406677 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.406685 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.406689 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.406809 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.407196 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.407203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.407207 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.407533 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.407690 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.408058 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.408256 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:12:31.408605 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.395092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60570)
0925 15:12:31.395310 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.395332 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.395468 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.395836 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.395841 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.396330 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.396569 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.396581 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.397721 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.397727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.399100 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.399114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.399804 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.399809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.400042 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.400624 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.400648 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.403124 (+ 2476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.406149 (+ 3025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.406156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.406161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.406557 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.406931 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.406936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.406941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.407089 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.407655 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.407660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.407819 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.407958 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.408236 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 15:12:31.409615 27344 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:31.410615 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.410185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36910)
0925 15:12:31.410360 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.410455 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:12:31.411549 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.411190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36918)
0925 15:12:31.411329 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.411406 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 15:12:31.411734 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.419688 27418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 3
I20260925 15:12:31.420679 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.420222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36926)
0925 15:12:31.420415 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.420520 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 15:12:31.421809 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.421252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36934)
0925 15:12:31.421499 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.421641 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260925 15:12:31.422036 27418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.462960 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 1
I20260925 15:12:31.464188 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.463644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36948)
0925 15:12:31.463887 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.463996 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 15:12:31.465179 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.464827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36950)
0925 15:12:31.464973 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.465056 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 15:12:31.465399 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
W20260925 15:12:31.525038 27425 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:31.525301 27425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:31.525360 27425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:31.525434 27425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:31.531883 27425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:31.531949 27425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:31.532018 27425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:31.532054 27425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:31.532106 27425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:31.533640 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 2
I20260925 15:12:31.534789 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.534237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36956)
0925 15:12:31.534463 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.534582 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:12:31.535775 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.535403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36966)
0925 15:12:31.535557 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.535637 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:12:31.535988 27344 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.541800 27425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:39097,127.4.205.253:45271,127.4.205.252:39429
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:39429
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.205.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:31.542842 27425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:31.543988 27425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:31.554911 27430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:31.554877 27431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:31.554858 27433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:31.558802 27425 server_base.cc:1034] running on GCE node
I20260925 15:12:31.559453 27425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:31.560726 27425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:31.561959 27425 hybrid_clock.cc:648] HybridClock initialized: now 1790349151561908 us; error 45 us; skew 500 ppm
I20260925 15:12:31.562274 27425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:31.565256 27418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 4
I20260925 15:12:31.565357 27425 init.cc:377] Logged in from keytab as kudu/127.4.205.252@KRBTEST.COM (short username kudu)
I20260925 15:12:31.566375 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.565933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36968)
0925 15:12:31.566158 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.566242 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260925 15:12:31.566989 27425 webserver.cc:460] Webserver started at http://127.4.205.252:45981/ using document root <none> and password file <none>
I20260925 15:12:31.567394 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.567060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:36984)
0925 15:12:31.567216 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.567296 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260925 15:12:31.567596 27418 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:39429: Network error: Client connection negotiation failed: client connection to 127.4.205.252:39429: connect: Connection refused (error 111)
I20260925 15:12:31.567776 27425 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:31.567888 27425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:31.568223 27425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:31.571105 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data/instance:
uuid: "8ddbe458cfa74a588d976a3d370ec4a5"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "d6b76de6bfb4ce12f90a0ec2affa163c"
server_key_iv: "97c563f42f3ecc03b1d4e325747c7c19"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.571815 27425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal/instance:
uuid: "8ddbe458cfa74a588d976a3d370ec4a5"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "d6b76de6bfb4ce12f90a0ec2affa163c"
server_key_iv: "97c563f42f3ecc03b1d4e325747c7c19"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.576412 27425 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 15:12:31.579434 27440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:31.580654 27425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:12:31.580818 27425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "8ddbe458cfa74a588d976a3d370ec4a5"
format_stamp: "Formatted at 2026-09-25 15:12:31 on dist-test-slave-64fx"
server_key: "d6b76de6bfb4ce12f90a0ec2affa163c"
server_key_iv: "97c563f42f3ecc03b1d4e325747c7c19"
server_key_version: "encryptionkey@0"
I20260925 15:12:31.581058 27425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:31.595160 27425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:31.599010 27425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:31.599243 27425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:31.616326 27425 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.252:39429
I20260925 15:12:31.616355 27491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.252:39429 every 8 connection(s)
I20260925 15:12:31.617503 27425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
I20260925 15:12:31.620918 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27425
I20260925 15:12:31.621101 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348808911781-4919-0/minicluster-data/master-2/wal/instance
I20260925 15:12:31.621646 4919 external_mini_cluster.cc:1428] Setting key fc9d47cc959ee438d32024e885d03c16
I20260925 15:12:31.621701 27492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:31.630645 27492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:31.642172 27344 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 3
I20260925 15:12:31.644786 27495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.623321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.4.205.252:39429)
0925 15:12:31.623993 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.624002 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.624038 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:12:31.626151 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.626170 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.626492 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.626777 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.628361 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.629521 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.631318 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.631960 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.632066 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.636334 (+ 4268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.636340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.636349 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.636377 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.639950 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.640527 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.640536 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.640545 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.640745 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.641180 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.641183 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.641185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.641732 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.641953 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.642476 (+ 523us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.643001 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":317,"threads_started":1}
I20260925 15:12:31.648625 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.633202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:39097 (local address 127.0.0.1:37172)
0925 15:12:31.633869 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.633903 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.634040 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.634407 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.634411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.634942 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.635161 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.635168 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.636528 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.636533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.638280 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.638296 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.638990 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.638996 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.639170 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.639744 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.639779 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.642244 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.645932 (+ 3688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.645943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.645950 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.646359 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.646754 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.646760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.646765 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.646928 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.647608 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.647615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.647811 (+ 196us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.647988 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.648250 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":331,"threads_started":1}
I20260925 15:12:31.649036 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.633228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37172 (local address 127.4.205.254:39097)
0925 15:12:31.633444 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.633450 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.634002 (+ 552us) server_negotiation.cc:407] Connection header received
0925 15:12:31.634167 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.634172 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.634246 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.634368 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.635336 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.636366 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.638410 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.638816 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.638907 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.642472 (+ 3565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.642481 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.642487 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.642533 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.645786 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.646481 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.646487 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.646492 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.646643 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.647063 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.647069 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.647072 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.647394 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.647542 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.648187 (+ 645us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.648480 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 15:12:31.651224 27492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:31.660526 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.642707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:37002)
0925 15:12:31.642928 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.642946 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.643060 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.644439 (+ 1379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.644446 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.644981 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.645218 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.645231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.647529 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.647533 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.649071 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.649082 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.650362 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.650368 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.650563 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.651066 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.651093 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.653676 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.657746 (+ 4070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.657752 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.657758 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.658301 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.658741 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.658746 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.658750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.658900 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.659520 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.659525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.659704 (+ 179us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.659853 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.660122 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 15:12:31.661717 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.643309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37002 (local address 127.4.205.252:39429)
0925 15:12:31.643875 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.643881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.643901 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:31.643946 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.643968 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.644284 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.644386 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.645386 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.647364 (+ 1978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.649637 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.650222 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.650351 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.653973 (+ 3622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.653984 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.653988 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.654023 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.657591 (+ 3568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.658472 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.658478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.658483 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.658618 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.659062 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.659068 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.659073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.659384 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.660847 (+ 1463us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.661005 (+ 158us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.661205 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":301,"threads_started":1}
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:31.667975 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.647595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.4.205.254:39097)
0925 15:12:31.649309 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.649317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.649338 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:12:31.649394 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.649401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.649479 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.649631 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.651452 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.652613 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.654378 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.655391 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.655521 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.660641 (+ 5120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.660648 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.660653 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.660686 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.664741 (+ 4055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.665493 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.665499 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.665502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.665662 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.666125 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.666131 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.666135 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.666524 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.666807 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.667195 (+ 388us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.667425 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1589,"thread_start_us":17433,"threads_started":1}
I20260925 15:12:31.667971 27344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995: Bootstrap starting.
I20260925 15:12:31.671559 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.651956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:45271 (local address 127.0.0.1:60574)
0925 15:12:31.653763 (+ 1807us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.653785 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.653971 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.654602 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.654609 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.655233 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.655501 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.655519 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.657312 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.657317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.659153 (+ 1836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.659171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.660906 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.660932 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.661210 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.662369 (+ 1159us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.662399 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.665089 (+ 2690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.668893 (+ 3804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.668899 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.668903 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.669358 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.669841 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.669843 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.669846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.669990 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.670686 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.670690 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.670803 (+ 113us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.670941 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.671223 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1661}
I20260925 15:12:31.671924 27422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.652472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60574 (local address 127.4.205.253:45271)
0925 15:12:31.652694 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.652701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.653865 (+ 1164us) server_negotiation.cc:407] Connection header received
0925 15:12:31.654147 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.654154 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.654270 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.654353 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.655727 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.657092 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.659348 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.660192 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.660305 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.665256 (+ 4951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.665261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.665264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.665294 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.668763 (+ 3469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.669499 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.669505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.669510 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.669726 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.670120 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.670126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.670130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.670473 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.670623 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.671053 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.671267 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:12:31.672389 27344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:31.673144 27492 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:31.673722 27344 log.cc:824] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:31.676674 27344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995: No bootstrap required, opened a new log
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349150, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:31.685444 27344 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.685777 27344 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d52a2cee8324b268190f8cd79419995, State: Initialized, Role: FOLLOWER
I20260925 15:12:31.686250 27344 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.687570 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.688032 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.688560 27496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.673789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:37018)
0925 15:12:31.674096 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.674112 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.674208 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.674576 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.674581 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.675014 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.675169 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.675180 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.676473 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.676478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.677482 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.677493 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.678548 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.678555 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.678736 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.679262 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.679286 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.681983 (+ 2697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.685964 (+ 3981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.685969 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.685972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.686493 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.686923 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.686927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.686929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.687088 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.687642 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.687646 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.687776 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.687901 (+ 125us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.688209 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260925 15:12:31.689553 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.673946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37018 (local address 127.4.205.252:39429)
0925 15:12:31.674097 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.674103 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.674158 (+ 55us) server_negotiation.cc:407] Connection header received
0925 15:12:31.674323 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.674330 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.674418 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.674544 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.675313 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.676348 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.677693 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.678363 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.678522 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.682170 (+ 3648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.682177 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.682183 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.682215 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.685815 (+ 3600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.686629 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.686633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.686637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.686783 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.687219 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.687223 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.687225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.687525 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.688898 (+ 1373us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.689026 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.689249 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:12:31.689980 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.669842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60586 (local address 127.4.205.253:45271)
0925 15:12:31.672240 (+ 2398us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.672246 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.672267 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:12:31.672313 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.672321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.672389 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.672510 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.674096 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.675303 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.676991 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.677617 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.677730 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.683116 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.683124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.683129 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.683163 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.686638 (+ 3475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.687470 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.687476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.687480 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.687630 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.688241 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.688247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.688252 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.688499 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.688810 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.689285 (+ 475us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.689501 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2253,"thread_start_us":2087,"threads_started":1}
I20260925 15:12:31.694636 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:12:31.695379 27492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: Bootstrap starting.
I20260925 15:12:31.697820 27319 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2d52a2cee8324b268190f8cd79419995"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37174
I20260925 15:12:31.698585 27492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:31.699530 27492 log.cc:824] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:31.700286 27344 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:31.700412 27319 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:12:31.700627 27319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:31.702584 27492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: No bootstrap required, opened a new log
I20260925 15:12:31.703987 27319 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.705295 27319 leader_election.cc:288] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [CANDIDATE]: Term 1 election: Requested vote from peers f802036a84614136bb1fddb5d210e372 (127.4.205.253:45271), 8ddbe458cfa74a588d976a3d370ec4a5 (127.4.205.252:39429)
I20260925 15:12:31.714936 27492 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.715410 27492 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddbe458cfa74a588d976a3d370ec4a5, State: Initialized, Role: FOLLOWER
I20260925 15:12:31.715997 27492 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.718099 27516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.719230 27492 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:31.719363 27467 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d52a2cee8324b268190f8cd79419995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ddbe458cfa74a588d976a3d370ec4a5"
I20260925 15:12:31.719964 27467 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:31.727038 27467 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52a2cee8324b268190f8cd79419995 in term 1.
I20260925 15:12:31.728482 27294 leader_election.cc:302] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2d52a2cee8324b268190f8cd79419995, 8ddbe458cfa74a588d976a3d370ec4a5; no voters:
I20260925 15:12:31.728948 27501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:31.733283 27516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.744405 27518 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:12:31.750088 27501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [term 1 LEADER]: Becoming Leader. State: Replica: 2d52a2cee8324b268190f8cd79419995, State: Running, Role: LEADER
I20260925 15:12:31.750810 27501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.754525 27347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.724575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37186 (local address 127.4.205.254:39097)
0925 15:12:31.724799 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.724813 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.727264 (+ 2451us) server_negotiation.cc:407] Connection header received
0925 15:12:31.728174 (+ 910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.728181 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.728265 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.728333 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.733109 (+ 4776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.734484 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.739193 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.740069 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.740180 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.745207 (+ 5027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.745213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.745217 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.745254 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.750872 (+ 5618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.751906 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.751914 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.751919 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.752112 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.752579 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.752584 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.752590 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.752933 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.753181 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.753858 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.754144 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:12:31.757138 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.724938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.4.205.253:45271)
0925 15:12:31.725123 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.725132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.725821 (+ 689us) server_negotiation.cc:407] Connection header received
0925 15:12:31.726105 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.726113 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.726201 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.726260 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.734935 (+ 8675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.736326 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.742315 (+ 5989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.743201 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.743316 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.749274 (+ 5958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.749280 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.749287 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.749326 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.753590 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.754998 (+ 1408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.755028 (+ 30us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.755032 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.755239 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.755664 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.755670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.755689 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.756028 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.756260 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.756602 (+ 342us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.756810 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:12:31.765043 27524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d52a2cee8324b268190f8cd79419995. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.765822 27524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:31.769310 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.726250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37032 (local address 127.4.205.252:39429)
0925 15:12:31.726426 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.726434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.745097 (+ 18663us) server_negotiation.cc:407] Connection header received
0925 15:12:31.746043 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.746050 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.746143 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.746222 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.753967 (+ 7745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.755363 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.757812 (+ 2449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.758482 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.758570 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.762380 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.762387 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.762392 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.762431 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.766156 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.766894 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.766900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.766904 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.767068 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.767444 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.767449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.767453 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.767794 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.768097 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.768543 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.768871 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:12:31.766834 27277 master_runner.cc:186] Error getting master registration for 127.4.205.253:45271: OK, instance_id { permanent_uuid: "f802036a84614136bb1fddb5d210e372" instance_seqno: 1790349151147062 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260925 15:12:31.777659 27534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:12:31.778860 27534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:12:31.779873 27277 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:12:31.780016 27536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:31.788234 27536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:31.779580 27425 master_runner.cc:186] Error getting master registration for 127.4.205.253:45271: OK, instance_id { permanent_uuid: "f802036a84614136bb1fddb5d210e372" instance_seqno: 1790349151147062 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:12:31.794965 27425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:12:31.807113 27467 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [term 1 FOLLOWER]: Refusing update from remote peer 2d52a2cee8324b268190f8cd79419995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:31.807972 27524 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:31.828460 27516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d52a2cee8324b268190f8cd79419995. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.828778 27516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.839401 27538 mvcc.cc:205] Tried to move back new op lower bound from 7333270125793214464 to 7333270125577949184. Current Snapshot: MvccSnapshot[applied={T|T < 7333270125793214464}]
I20260925 15:12:31.843520 27516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.843756 27516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.849295 27536 catalog_manager.cc:1347] Generated new cluster ID: 0e90d6b839cf4dfab67f20dd0f9354bc
I20260925 15:12:31.849392 27536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:31.850553 27524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.850814 27524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:31.866736 27418 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } attempt: 5
I20260925 15:12:31.874476 27536 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 15:12:31 dist-test-slave-64fx krb5kdc[27264](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349151, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:12:31.878167 27536 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:31.885317 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.867427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.4.205.252:39429)
0925 15:12:31.867569 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:31.867575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:31.867666 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:12:31.867837 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:31.867843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:31.867911 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:31.867994 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:31.868925 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.870126 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.872117 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.872820 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.872926 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.877663 (+ 4737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:31.877687 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:31.877691 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:31.877728 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:31.881545 (+ 3817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.882366 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.882371 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.882374 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.882513 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:31.882924 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:31.882930 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:31.882933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:31.883254 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:31.883388 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:31.884673 (+ 1285us) server_negotiation.cc:299] Negotiation successful
0925 15:12:31.884913 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:12:31.886302 27421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:31.867235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:39429 (local address 127.0.0.1:37040)
0925 15:12:31.867586 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:31.867607 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:31.867713 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:31.868014 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:31.868018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:31.868588 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:31.868798 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.868810 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.870304 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.870309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.871609 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:31.871623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:31.873349 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:31.873357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:31.873596 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:31.874493 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:31.874519 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:31.877435 (+ 2916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:31.881723 (+ 4288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.881728 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.881732 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.882237 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.882646 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:31.882650 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:31.882654 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:31.882809 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:31.883446 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:31.883451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:31.883645 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 15:12:31.883792 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 15:12:31.885919 (+ 2127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245}
I20260925 15:12:31.891790 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: Bootstrap starting.
I20260925 15:12:31.895964 27418 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:31.897284 27418 log.cc:824] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:31.909132 27536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995: Generated new TSK 0
I20260925 15:12:31.910965 27536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:31.911214 27418 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: No bootstrap required, opened a new log
I20260925 15:12:31.922781 27418 raft_consensus.cc:348] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.923259 27418 raft_consensus.cc:729] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f802036a84614136bb1fddb5d210e372, State: Initialized, Role: FOLLOWER
I20260925 15:12:31.923769 27418 consensus_queue.cc:260] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } }
I20260925 15:12:31.926170 27418 sys_catalog.cc:564] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:31.943146 27392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:31.947932 27392 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [term 1 FOLLOWER]: Refusing update from remote peer 2d52a2cee8324b268190f8cd79419995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:31.949146 27501 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:31.929870 27393 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2d52a2cee8324b268190f8cd79419995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "f802036a84614136bb1fddb5d210e372"
I20260925 15:12:31.955745 27393 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2d52a2cee8324b268190f8cd79419995 in term 1.
I20260925 15:12:31.925019 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.962569 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.997972 27544 sys_catalog.cc:455] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d52a2cee8324b268190f8cd79419995. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.998322 27544 sys_catalog.cc:458] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:31.998561 27550 sys_catalog.cc:455] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:31.998945 27550 sys_catalog.cc:458] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:32.032838 27561 catalog_manager.cc:1259] Loaded cluster ID: 0e90d6b839cf4dfab67f20dd0f9354bc
I20260925 15:12:32.032946 27561 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: loading cluster ID for follower catalog manager: success
I20260925 15:12:32.036949 27561 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: acquiring CA information for follower catalog manager: success
I20260925 15:12:32.039321 27561 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f802036a84614136bb1fddb5d210e372: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 15:12:32.045733 4919 mini_kdc.cc:348] Time spent kdestroy: real 0.120s user 0.008s sys 0.014s
W20260925 15:12:32.093186 27347 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.079244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.4.205.254:39097)
0925 15:12:32.079453 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.079459 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.090089 (+ 10630us) server_negotiation.cc:407] Connection header received
0925 15:12:32.091640 (+ 1551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.091647 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.091737 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.091819 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.092933 (+ 1114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37196: BlockingRecv error: recv got EOF from 127.0.0.1:37196 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
W20260925 15:12:32.095261 27499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.076184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60602 (local address 127.4.205.253:45271)
0925 15:12:32.076396 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.076406 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.089631 (+ 13225us) server_negotiation.cc:407] Connection header received
0925 15:12:32.093777 (+ 4146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.093783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.093867 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.093952 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.094994 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60602: BlockingRecv error: recv got EOF from 127.0.0.1:60602 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260925 15:12:32.099051 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.090707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.4.205.252:39429)
0925 15:12:32.090878 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.090884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.096683 (+ 5799us) server_negotiation.cc:407] Connection header received
0925 15:12:32.097733 (+ 1050us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.097738 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.097824 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.097897 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.098782 (+ 885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37048: BlockingRecv error: recv got EOF from 127.0.0.1:37048 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 15:12:32.146996 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.128774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60618 (local address 127.4.205.253:45271)
0925 15:12:32.128949 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.128955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.139469 (+ 10514us) server_negotiation.cc:407] Connection header received
0925 15:12:32.139911 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.139918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.140099 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.140218 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.140879 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.142095 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.144060 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.144798 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.144888 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.145578 (+ 690us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.146195 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.146299 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.146544 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.146627 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:12:32.199831 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.186461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37056 (local address 127.4.205.252:39429)
0925 15:12:32.186635 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.186642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.198058 (+ 11416us) server_negotiation.cc:407] Connection header received
0925 15:12:32.198346 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.198355 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.198481 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.198556 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.199589 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37056: BlockingRecv error: recv got EOF from 127.0.0.1:37056 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:12:32.251792 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.234373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.4.205.253:45271)
0925 15:12:32.234559 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.234565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.245093 (+ 10528us) server_negotiation.cc:407] Connection header received
0925 15:12:32.245593 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.245602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.245697 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.245762 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.246421 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.247556 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.249584 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.250304 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.250436 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.251025 (+ 589us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.251150 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.251204 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.251405 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.251487 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 15:12:32.296310 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.283651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37062 (local address 127.4.205.252:39429)
0925 15:12:32.283837 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.283843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.294432 (+ 10589us) server_negotiation.cc:407] Connection header received
0925 15:12:32.294890 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.294896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.295013 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.295089 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.296095 (+ 1006us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37062: BlockingRecv error: recv got EOF from 127.0.0.1:37062 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:12:32.355374 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.336632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60628 (local address 127.4.205.253:45271)
0925 15:12:32.336884 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.336893 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.350138 (+ 13245us) server_negotiation.cc:407] Connection header received
0925 15:12:32.350379 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.350384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.350458 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.350570 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.351068 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.352219 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.353427 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.354130 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.354255 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.354573 (+ 318us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.354727 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.354821 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.355000 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.355084 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260925 15:12:32.362862 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.360771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.4.205.252:39429)
0925 15:12:32.360980 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.360988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.361459 (+ 471us) server_negotiation.cc:407] Connection header received
0925 15:12:32.361672 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.361676 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.361744 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.361826 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.362593 (+ 767us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37074: BlockingRecv error: recv got EOF from 127.0.0.1:37074 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 15:12:32.375842 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.370134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60642 (local address 127.4.205.253:45271)
0925 15:12:32.370328 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.370333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.370637 (+ 304us) server_negotiation.cc:407] Connection header received
0925 15:12:32.370810 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.370813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.370869 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.370936 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.371514 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.372330 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.374254 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.374663 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.374790 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.375186 (+ 396us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.375284 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.375351 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.375500 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.375556 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 15:12:32.383705 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.382020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37090 (local address 127.4.205.252:39429)
0925 15:12:32.382295 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.382302 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.382497 (+ 195us) server_negotiation.cc:407] Connection header received
0925 15:12:32.382646 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.382649 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.382712 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.382787 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.383501 (+ 714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37090: BlockingRecv error: recv got EOF from 127.0.0.1:37090 (error 108)
Metrics: {"server-negotiator.queue_time_us":173}
I20260925 15:12:32.401420 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.394445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60654 (local address 127.4.205.253:45271)
0925 15:12:32.394577 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.394581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.394886 (+ 305us) server_negotiation.cc:407] Connection header received
0925 15:12:32.395052 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.395055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.395127 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.395197 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.395688 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.397025 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.398925 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.399700 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.399825 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.400400 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.400535 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.400638 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.400979 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.401062 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260925 15:12:32.410519 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.409028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56812 (local address 127.4.205.252:39429)
0925 15:12:32.409256 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.409263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.409283 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:32.409392 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.409396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.409457 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.409533 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.410327 (+ 794us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56812: BlockingRecv error: recv got EOF from 127.0.0.1:56812 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:12:32.434720 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.429302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43224 (local address 127.4.205.253:45271)
0925 15:12:32.429508 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.429516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.429957 (+ 441us) server_negotiation.cc:407] Connection header received
0925 15:12:32.430069 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.430073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.430146 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.430241 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.430584 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.431586 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.432861 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.433634 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.433758 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.434090 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.434221 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.434315 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.434409 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.434488 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260925 15:12:32.442497 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.440458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56816 (local address 127.4.205.252:39429)
0925 15:12:32.440667 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.440672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.441009 (+ 337us) server_negotiation.cc:407] Connection header received
0925 15:12:32.441180 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.441184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.441252 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.441329 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.442249 (+ 920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56816: BlockingRecv error: recv got EOF from 127.0.0.1:56816 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 15:12:32.481350 27501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2d52a2cee8324b268190f8cd79419995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2d52a2cee8324b268190f8cd79419995" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 39097 } } peers { permanent_uuid: "f802036a84614136bb1fddb5d210e372" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 45271 } } peers { permanent_uuid: "8ddbe458cfa74a588d976a3d370ec4a5" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 39429 } } }
I20260925 15:12:32.481688 27501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d52a2cee8324b268190f8cd79419995 [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:32.486541 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.478463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43226 (local address 127.4.205.253:45271)
0925 15:12:32.478651 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.478659 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.479118 (+ 459us) server_negotiation.cc:407] Connection header received
0925 15:12:32.479307 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.479311 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.479420 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.479496 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.479949 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.480827 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.483586 (+ 2759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.484464 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.484575 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.485434 (+ 859us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.485608 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.485684 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.485961 (+ 277us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.486099 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260925 15:12:32.494656 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.492684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56818 (local address 127.4.205.252:39429)
0925 15:12:32.492889 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.492894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.493219 (+ 325us) server_negotiation.cc:407] Connection header received
0925 15:12:32.493414 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.493423 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.493498 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.493658 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.494422 (+ 764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56818: BlockingRecv error: recv got EOF from 127.0.0.1:56818 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:12:32.567512 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.561817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43230 (local address 127.4.205.253:45271)
0925 15:12:32.562061 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.562068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.562436 (+ 368us) server_negotiation.cc:407] Connection header received
0925 15:12:32.562593 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.562602 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.562706 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.562810 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.563159 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.564262 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.565578 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.566264 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.566414 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.566730 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.566880 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.566961 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.567085 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.567158 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:12:32.574564 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.572729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.4.205.252:39429)
0925 15:12:32.572929 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.572934 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.573215 (+ 281us) server_negotiation.cc:407] Connection header received
0925 15:12:32.573368 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.573372 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.573437 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.573513 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.574325 (+ 812us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56830: BlockingRecv error: recv got EOF from 127.0.0.1:56830 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 15:12:32.711236 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.705710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43238 (local address 127.4.205.253:45271)
0925 15:12:32.705936 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.705943 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.706223 (+ 280us) server_negotiation.cc:407] Connection header received
0925 15:12:32.706377 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.706381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.706455 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.706529 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.706855 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.707710 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.709351 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.709990 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.710112 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.710530 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.710620 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.710731 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.710858 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.710946 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 15:12:32.717765 27497 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:32.716191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56844 (local address 127.4.205.252:39429)
0925 15:12:32.716363 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.716367 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.716699 (+ 332us) server_negotiation.cc:407] Connection header received
0925 15:12:32.716827 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.716830 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.716896 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.716975 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:32.717533 (+ 558us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56844: BlockingRecv error: recv got EOF from 127.0.0.1:56844 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:12:32.780642 27534 catalog_manager.cc:1259] Loaded cluster ID: 0e90d6b839cf4dfab67f20dd0f9354bc
I20260925 15:12:32.780766 27534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: loading cluster ID for follower catalog manager: success
I20260925 15:12:32.783505 27534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: acquiring CA information for follower catalog manager: success
I20260925 15:12:32.785284 27534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ddbe458cfa74a588d976a3d370ec4a5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:12:32.985919 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.981212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43248 (local address 127.4.205.253:45271)
0925 15:12:32.981388 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.981393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.981747 (+ 354us) server_negotiation.cc:407] Connection header received
0925 15:12:32.981898 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.981902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.981969 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.982075 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.982395 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.983107 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.984430 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.984806 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.984986 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.985348 (+ 362us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.985438 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.985507 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.985642 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.985699 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 15:12:32.996412 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:32.990976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56858 (local address 127.4.205.252:39429)
0925 15:12:32.991134 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:32.991138 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:32.991454 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:12:32.991584 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:32.991588 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:32.991747 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:32.991885 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:32.992255 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.992915 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.994372 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:32.994884 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:32.994978 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:32.995381 (+ 403us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:32.995892 (+ 511us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:32.996073 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:32.996122 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:12:32.996181 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:33.011250 27497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:33.004010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.4.205.252:39429)
0925 15:12:33.004230 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:33.004237 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:33.004927 (+ 690us) server_negotiation.cc:407] Connection header received
0925 15:12:33.005170 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:33.005175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:33.005253 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:33.005341 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:33.005983 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.007049 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.009197 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.009844 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.009954 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:33.010519 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:33.010633 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:33.010715 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:33.010872 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:12:33.010968 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:12:33.011273 27499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:33.002308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43250 (local address 127.4.205.253:45271)
0925 15:12:33.002496 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:33.002505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:33.003846 (+ 1341us) server_negotiation.cc:407] Connection header received
0925 15:12:33.004116 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:33.004126 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:33.004235 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:33.004305 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:33.005067 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.006004 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.007881 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.008489 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.008608 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:33.009336 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:33.009451 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:33.009547 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:33.009743 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:12:33.010972 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 15:12:33.012964 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:33.002222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52608 (local address 127.4.205.254:39097)
0925 15:12:33.005487 (+ 3265us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:33.005492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:33.005509 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:12:33.005654 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:33.005659 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:33.005881 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:33.006096 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:33.006693 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.007597 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.009785 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:33.010454 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:33.010588 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:33.011266 (+ 678us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:33.011744 (+ 478us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:33.011849 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:33.012576 (+ 727us) server_negotiation.cc:299] Negotiation successful
0925 15:12:33.012654 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3155,"thread_start_us":3074,"threads_started":1}
I20260925 15:12:33.017853 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27277
I20260925 15:12:33.018754 27339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:33.129361 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27277
I20260925 15:12:33.149767 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27351
I20260925 15:12:33.150911 27413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:33.255038 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27351
I20260925 15:12:33.273710 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27425
I20260925 15:12:33.274688 27487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:33.376855 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27425
2026-09-25T15:12:33Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4422 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:12:33 dist-test-slave-64fx krb5kdc[27687](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:12:33 dist-test-slave-64fx krb5kdc[27687](info): set up 2 sockets
Sep 25 15:12:33 dist-test-slave-64fx krb5kdc[27687](info): commencing operation
krb5kdc: starting...
W20260925 15:12:34.450160 4919 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/test-user.keytab.
Sep 25 15:12:34 dist-test-slave-64fx krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:12:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:12:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.4.205.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
Entry for principal kudu/127.4.205.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab.
I20260925 15:12:34.677435 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41345
--webserver_interface=127.4.205.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:12:34.983856 27700 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:34.984230 27700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:34.984369 27700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:34.984498 27700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:34.991679 27700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:34.991748 27700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:34.991815 27700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:34.991876 27700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:34.991973 27700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:35.001312 27700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.254.keytab
--principal=kudu/127.4.205.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.254:41345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.4.205.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:35.002456 27700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:35.003662 27700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:35.014508 27706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.014544 27708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.014564 27705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:35.015563 27700 server_base.cc:1034] running on GCE node
I20260925 15:12:35.016206 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:35.017477 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:35.018682 27700 hybrid_clock.cc:648] HybridClock initialized: now 1790349155018630 us; error 24 us; skew 500 ppm
I20260925 15:12:35.018956 27700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:35.028321 27700 init.cc:377] Logged in from keytab as kudu/127.4.205.254@KRBTEST.COM (short username kudu)
I20260925 15:12:35.029728 27700 webserver.cc:460] Webserver started at http://127.4.205.254:42761/ using document root <none> and password file <none>
I20260925 15:12:35.030427 27700 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:35.030539 27700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:35.030857 27700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:35.033690 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data/instance:
uuid: "0d5d4f2612ec46b1bffe71275ecaf44f"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "27addef4489e5ce5489069ce5b2a6b56"
server_key_iv: "73db1ad5834cb16b72bb59e89d0e3e20"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.034438 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance:
uuid: "0d5d4f2612ec46b1bffe71275ecaf44f"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "27addef4489e5ce5489069ce5b2a6b56"
server_key_iv: "73db1ad5834cb16b72bb59e89d0e3e20"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.039325 27700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 15:12:35.042510 27715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:35.043763 27700 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260925 15:12:35.043973 27700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal
uuid: "0d5d4f2612ec46b1bffe71275ecaf44f"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "27addef4489e5ce5489069ce5b2a6b56"
server_key_iv: "73db1ad5834cb16b72bb59e89d0e3e20"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.044183 27700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:35.059514 27700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:35.063009 27700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:35.063261 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:35.080366 27700 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.254:41345
I20260925 15:12:35.080382 27766 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.254:41345 every 8 connection(s)
I20260925 15:12:35.081545 27700 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/data/info.pb
I20260925 15:12:35.084641 27767 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:35.088105 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27700
I20260925 15:12:35.088287 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-0/wal/instance
I20260925 15:12:35.088668 4919 external_mini_cluster.cc:1428] Setting key 0d87f4de62b476cf62ba43e47100417c
I20260925 15:12:35.090679 27767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.253@KRBTEST.COM; defaulting to no policy
I20260925 15:12:35.115298 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.093920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41345 (local address 127.0.0.1:47268)
0925 15:12:35.094763 (+ 843us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.094814 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:35.097696 (+ 2882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:35.098549 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:35.098555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:35.099329 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:35.099834 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.099859 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.102544 (+ 2685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.102551 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.104439 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.104454 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.105356 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.105363 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.105630 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.106280 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:35.106300 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:35.108535 (+ 2235us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:35.111607 (+ 3072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.111618 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.111629 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.111982 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.112330 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.112336 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.112342 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.112483 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.113122 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:35.113130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:35.113317 (+ 187us) client_negotiation.cc:769] Sending connection context
0925 15:12:35.113529 (+ 212us) client_negotiation.cc:240] Negotiation successful
0925 15:12:35.114085 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"mutex_wait_us":198,"thread_start_us":326,"threads_started":1}
I20260925 15:12:35.115321 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.093920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47268 (local address 127.4.205.254:41345)
0925 15:12:35.097445 (+ 3525us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:35.097454 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:35.097505 (+ 51us) server_negotiation.cc:407] Connection header received
0925 15:12:35.098075 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:35.098083 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:35.098307 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:35.098525 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:35.100122 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.102362 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.104596 (+ 2234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.105128 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.105311 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.108740 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:35.108747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:35.108757 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:35.108781 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:35.111494 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.112096 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.112109 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.112112 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.112233 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.112582 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.112589 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.112592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.112994 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:35.113177 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:35.113650 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 15:12:35.113978 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":3313,"spinlock_wait_cycles":20224,"thread_start_us":3073,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.253@KRBTEST.COM" created.
I20260925 15:12:35.118044 27767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:35.118993 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.118634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33568)
0925 15:12:35.118782 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.118889 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:12:35.119187 27719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
I20260925 15:12:35.120623 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.120241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33570)
0925 15:12:35.120412 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.120478 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 15:12:35.120858 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:33961: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
Entry for principal kudu/127.4.205.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab.
I20260925 15:12:35.142009 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:33961
--webserver_interface=127.4.205.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:12:35.170233 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } attempt: 1
I20260925 15:12:35.171309 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.170890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33576)
0925 15:12:35.171074 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.171189 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260925 15:12:35.172324 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.171985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33590)
0925 15:12:35.172117 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.172184 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 15:12:35.172587 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:33961: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
I20260925 15:12:35.240823 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } attempt: 2
I20260925 15:12:35.241989 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.241524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33594)
0925 15:12:35.241767 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.241854 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:12:35.242934 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.242585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33596)
0925 15:12:35.242741 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.242807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 15:12:35.243098 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:33961: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
I20260925 15:12:35.334326 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } attempt: 3
I20260925 15:12:35.335644 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.334987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33612)
0925 15:12:35.335287 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.335420 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260925 15:12:35.336752 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.336338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33620)
0925 15:12:35.336509 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.336594 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:12:35.336941 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:33961: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
W20260925 15:12:35.444607 27774 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:35.444904 27774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:35.444988 27774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:35.445087 27774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:35.451970 27774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:35.452064 27774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:35.452116 27774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:35.452162 27774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:35.452212 27774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:35.461110 27774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.253.keytab
--principal=kudu/127.4.205.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.253:33961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.4.205.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:35.462225 27774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:35.463913 27774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:12:35.474691 27779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.474866 27780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.474833 27782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:35.475701 27774 server_base.cc:1034] running on GCE node
I20260925 15:12:35.476330 27774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:35.477501 27774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:35.478756 27774 hybrid_clock.cc:648] HybridClock initialized: now 1790349155478689 us; error 35 us; skew 500 ppm
I20260925 15:12:35.479061 27774 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 15:12:35.480154 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } attempt: 4
I20260925 15:12:35.481264 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.480770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33636)
0925 15:12:35.481061 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.481138 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:35.482234 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.481881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33648)
0925 15:12:35.482027 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.482121 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 15:12:35.482398 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.253:33961: Network error: Client connection negotiation failed: client connection to 127.4.205.253:33961: connect: Connection refused (error 111)
I20260925 15:12:35.482467 27774 init.cc:377] Logged in from keytab as kudu/127.4.205.253@KRBTEST.COM (short username kudu)
I20260925 15:12:35.483789 27774 webserver.cc:460] Webserver started at http://127.4.205.253:40691/ using document root <none> and password file <none>
I20260925 15:12:35.484428 27774 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:35.484548 27774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:35.484850 27774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:35.487617 27774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data/instance:
uuid: "9b0d3d5bd1624c04956f2c4ad45acdff"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "7804c2021f0bcd876047bcc6dff157be"
server_key_iv: "216a98ef158a28d41ad96540bd1d6a61"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.488345 27774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal/instance:
uuid: "9b0d3d5bd1624c04956f2c4ad45acdff"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "7804c2021f0bcd876047bcc6dff157be"
server_key_iv: "216a98ef158a28d41ad96540bd1d6a61"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.493194 27774 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 15:12:35.496344 27789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:35.497445 27774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:12:35.497637 27774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal
uuid: "9b0d3d5bd1624c04956f2c4ad45acdff"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "7804c2021f0bcd876047bcc6dff157be"
server_key_iv: "216a98ef158a28d41ad96540bd1d6a61"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.497826 27774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:35.516577 27774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:35.520180 27774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:35.520429 27774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:35.537926 27774 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.253:33961
I20260925 15:12:35.537973 27840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.253:33961 every 8 connection(s)
I20260925 15:12:35.539151 27774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/data/info.pb
I20260925 15:12:35.542543 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27774
I20260925 15:12:35.542630 27841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:35.542742 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-1/wal/instance
I20260925 15:12:35.543272 4919 external_mini_cluster.cc:1428] Setting key 522ee8283521e7ad4a6d96ecf5db7d94
I20260925 15:12:35.552040 27841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.4.205.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.4.205.252@KRBTEST.COM" created.
I20260925 15:12:35.572535 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.555105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47274 (local address 127.4.205.254:41345)
0925 15:12:35.555271 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:35.555275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:35.556493 (+ 1218us) server_negotiation.cc:407] Connection header received
0925 15:12:35.558266 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:35.558272 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:35.558357 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:35.558476 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:35.559796 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.560448 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.562008 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.562412 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.562551 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.566642 (+ 4091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:35.566647 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:35.566651 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:35.566681 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:35.569361 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.570092 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.570096 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.570099 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.570212 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.570567 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.570574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.570580 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.570892 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:35.571044 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:35.571865 (+ 821us) server_negotiation.cc:299] Negotiation successful
0925 15:12:35.572116 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 15:12:35.573123 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.555657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41345 (local address 127.0.0.1:47274)
0925 15:12:35.556348 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.556386 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:35.558023 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:35.558675 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:35.558683 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:35.559154 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:35.559636 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.559651 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.560605 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.560609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.561850 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.561860 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.562629 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.562635 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.562849 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.563511 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:35.563532 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:35.566407 (+ 2875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:35.569487 (+ 3080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.569494 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.569507 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.569979 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.570305 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.570308 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.570310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.570459 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.571001 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:35.571006 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:35.571449 (+ 443us) client_negotiation.cc:769] Sending connection context
0925 15:12:35.571709 (+ 260us) client_negotiation.cc:240] Negotiation successful
0925 15:12:35.572064 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":286,"threads_started":1}
I20260925 15:12:35.574947 27841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:35.589929 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.575917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33652 (local address 127.4.205.253:33961)
0925 15:12:35.576587 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:35.576593 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:35.576615 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:12:35.576652 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:35.576658 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:35.576815 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:35.576926 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:35.577850 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.578783 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.580126 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.581076 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.581219 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.584447 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:35.584457 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:35.584468 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:35.584493 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:35.587142 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.587914 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.587920 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.587928 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.588027 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.588439 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.588442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.588443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.588729 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:35.588873 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:35.589281 (+ 408us) server_negotiation.cc:299] Negotiation successful
0925 15:12:35.589651 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":132,"server-negotiator.queue_time_us":526,"thread_start_us":402,"threads_started":1}
I20260925 15:12:35.590001 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.575755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33652)
0925 15:12:35.575968 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.575992 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:35.576093 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:35.576978 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:35.576985 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:35.577459 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:35.577692 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.577704 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.578953 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.578960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.579999 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.580008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.581223 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.581227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.581339 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.581897 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:35.581914 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:35.584261 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:35.587266 (+ 3005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.587274 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.587279 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.587802 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.588132 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.588140 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.588146 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.588338 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.588854 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:35.588861 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:35.589016 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:12:35.589176 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 15:12:35.589488 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 15:12:35.592458 27841 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:35.593497 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.593012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50820)
0925 15:12:35.593194 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.593323 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:12:35.593762 27790 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.595315 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.594890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50834)
0925 15:12:35.595106 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.595194 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260925 15:12:35.595532 27841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
Entry for principal kudu/127.4.205.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab.
I20260925 15:12:35.604014 4919 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
/tmp/dist-test-taskSXSwaN/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:44009
--webserver_interface=127.4.205.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.4.205.212:41911
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:12:35.644871 27841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 1
I20260925 15:12:35.646209 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.645659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50836)
0925 15:12:35.645874 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.645999 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 15:12:35.647289 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.646848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50838)
0925 15:12:35.647025 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.647122 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 15:12:35.647509 27841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.715750 27841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 2
I20260925 15:12:35.716928 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.716436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50854)
0925 15:12:35.716646 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.716749 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 15:12:35.717916 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.717503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50870)
0925 15:12:35.717696 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.717773 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260925 15:12:35.718150 27841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.781658 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } attempt: 5
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:35.796681 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.782394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33654)
0925 15:12:35.782670 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.782697 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:35.782876 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:35.783252 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:35.783260 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:35.783883 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:35.784193 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.784210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.785173 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.785180 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.786894 (+ 1714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:35.786918 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.787692 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.787701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:35.787977 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.788739 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:35.788766 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:35.791436 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:35.794375 (+ 2939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.794381 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.794386 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.794727 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.795065 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:35.795090 (+ 25us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:35.795094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:35.795251 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:35.795756 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:35.795761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:35.795916 (+ 155us) client_negotiation.cc:769] Sending connection context
0925 15:12:35.796058 (+ 142us) client_negotiation.cc:240] Negotiation successful
0925 15:12:35.796339 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 15:12:35.796679 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.782674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.4.205.253:33961)
0925 15:12:35.782884 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:35.782888 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:35.782905 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:12:35.783015 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:35.783019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:35.783077 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:35.783156 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:35.784355 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.785010 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.787058 (+ 2048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:35.787509 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:35.787599 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:35.791630 (+ 4031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:35.791634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:35.791637 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:35.791666 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:35.794246 (+ 2580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.794834 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.794841 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.794845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.794966 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:35.795365 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:35.795371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:35.795377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:35.795642 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:35.795795 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:35.796160 (+ 365us) server_negotiation.cc:299] Negotiation successful
0925 15:12:35.796380 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 15:12:35.797614 27767 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:35.798599 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.798229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50876)
0925 15:12:35.798371 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.798467 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 15:12:35.799405 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.799122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50880)
0925 15:12:35.799253 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.799323 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260925 15:12:35.799606 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.809369 27841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 3
I20260925 15:12:35.810374 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.809931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50896)
0925 15:12:35.810130 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.810216 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 15:12:35.811318 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.810962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50904)
0925 15:12:35.811152 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.811224 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260925 15:12:35.811507 27841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.850788 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 1
I20260925 15:12:35.851926 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.851355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50918)
0925 15:12:35.851594 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.851720 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260925 15:12:35.852984 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.852593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50928)
0925 15:12:35.852755 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.852849 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 15:12:35.853210 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.921424 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 2
I20260925 15:12:35.922609 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.922089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50932)
0925 15:12:35.922318 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.922418 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260925 15:12:35.923544 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.923210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50944)
0925 15:12:35.923347 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.923423 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260925 15:12:35.923735 27767 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
W20260925 15:12:35.923998 27848 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:12:35.924294 27848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:12:35.924365 27848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:12:35.924446 27848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:12:35.931710 27848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:12:35.931794 27848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:12:35.931880 27848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:12:35.931922 27848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:12:35.931977 27848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:12:35.941727 27848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.205.212:41911
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.4.205.254:41345,127.4.205.253:33961,127.4.205.252:44009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/krb5kdc/kudu_127.4.205.252.keytab
--principal=kudu/127.4.205.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.205.252:44009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.4.205.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 14:12:43 UTC on ccycloud.quasar-rhlhux.root.comops.site
build id 4672728
ASAN enabled
I20260925 15:12:35.942920 27848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:12:35.944635 27848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:12:35.954681 27841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 4
I20260925 15:12:35.955857 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.955406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50958)
0925 15:12:35.955635 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.955712 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260925 15:12:35.956249 27853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.956403 27856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:12:35.956455 27854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:12:35.957078 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:35.956565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50970)
0925 15:12:35.956865 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:35.956943 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186}
W20260925 15:12:35.957278 27841 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.4.205.252:44009: Network error: Client connection negotiation failed: client connection to 127.4.205.252:44009: connect: Connection refused (error 111)
I20260925 15:12:35.957406 27848 server_base.cc:1034] running on GCE node
I20260925 15:12:35.958175 27848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:12:35.959465 27848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:12:35.960703 27848 hybrid_clock.cc:648] HybridClock initialized: now 1790349155960637 us; error 44 us; skew 500 ppm
I20260925 15:12:35.961067 27848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 15:12:35 dist-test-slave-64fx krb5kdc[27687](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:12:35.964146 27848 init.cc:377] Logged in from keytab as kudu/127.4.205.252@KRBTEST.COM (short username kudu)
I20260925 15:12:35.965649 27848 webserver.cc:460] Webserver started at http://127.4.205.252:45113/ using document root <none> and password file <none>
I20260925 15:12:35.966325 27848 fs_manager.cc:359] Metadata directory not provided
I20260925 15:12:35.966487 27848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:12:35.966813 27848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:12:35.969750 27848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data/instance:
uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "fce246e0c76d8b597b069eeacac790fc"
server_key_iv: "2cb214519bd3f5e4801fe2fc3afa8b99"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.970566 27848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal/instance:
uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "fce246e0c76d8b597b069eeacac790fc"
server_key_iv: "2cb214519bd3f5e4801fe2fc3afa8b99"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.975658 27848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260925 15:12:35.978958 27863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:12:35.980173 27848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:12:35.980376 27848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data,/tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal
uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9"
format_stamp: "Formatted at 2026-09-25 15:12:35 on dist-test-slave-64fx"
server_key: "fce246e0c76d8b597b069eeacac790fc"
server_key_iv: "2cb214519bd3f5e4801fe2fc3afa8b99"
server_key_version: "encryptionkey@0"
I20260925 15:12:35.980597 27848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:12:36.000651 27848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:12:36.004580 27848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:12:36.004834 27848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:12:36.022740 27848 rpc_server.cc:307] RPC server started. Bound to: 127.4.205.252:44009
I20260925 15:12:36.022766 27914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.205.252:44009 every 8 connection(s)
I20260925 15:12:36.023968 27848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/data/info.pb
I20260925 15:12:36.026273 4919 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu as pid 27848
I20260925 15:12:36.026448 4919 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskSXSwaN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348808911781-4919-0/minicluster-data/master-2/wal/instance
I20260925 15:12:36.026885 4919 external_mini_cluster.cc:1428] Setting key d6c86ccaed47a173512cb4c0e0edbad6
I20260925 15:12:36.027637 27915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:12:36.029920 27767 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 3
I20260925 15:12:36.038234 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.254@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:12:36.049911 27769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.030524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50988)
0925 15:12:36.030763 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:36.030783 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:36.030902 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:36.032435 (+ 1533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:36.032439 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:36.032956 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:36.033133 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.033143 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.035572 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.035579 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.037045 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.037061 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.038019 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.038025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.038214 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.038956 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:36.038979 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:36.042247 (+ 3268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:36.046863 (+ 4616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.046869 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.046874 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.047452 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.047973 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.047979 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.047984 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.048224 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.048855 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:36.048860 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:36.049038 (+ 178us) client_negotiation.cc:769] Sending connection context
0925 15:12:36.049189 (+ 151us) client_negotiation.cc:240] Negotiation successful
0925 15:12:36.049446 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:36.051330 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.030632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50988 (local address 127.4.205.252:44009)
0925 15:12:36.031284 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.031289 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.031309 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:36.031726 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.031740 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.032109 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.032458 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.033332 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.035412 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.037192 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.037857 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.037978 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.042526 (+ 4548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.042536 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.042549 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.042590 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.046761 (+ 4171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.047618 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.047623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.047627 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.047767 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.048318 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.048328 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.048332 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.048695 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.050285 (+ 1590us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.050394 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.050542 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":532,"thread_start_us":388,"threads_started":1}
I20260925 15:12:36.051383 27918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.028614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50972 (local address 127.4.205.252:44009)
0925 15:12:36.029409 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.029426 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.029476 (+ 50us) server_negotiation.cc:407] Connection header received
0925 15:12:36.031713 (+ 2237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.031738 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.032111 (+ 373us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.033299 (+ 1188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.034182 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.035412 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.036729 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.037512 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.037721 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.042590 (+ 4869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.042594 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.042597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.042628 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.046345 (+ 3717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.047215 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.047224 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.047232 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.047412 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.047869 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.047874 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.047877 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.048638 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.048961 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.049392 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.050124 (+ 732us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":563,"spinlock_wait_cycles":21376,"thread_start_us":347,"threads_started":1}
I20260925 15:12:36.058405 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.040795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.254:41345 (local address 127.0.0.1:47282)
0925 15:12:36.041402 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:36.041450 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:36.041671 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:36.042118 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:36.042123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:36.042946 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:36.043348 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.043362 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.044809 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.044815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.046705 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.046720 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.048002 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.048007 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.048179 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.048777 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:36.048797 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:36.051566 (+ 2769us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:36.055634 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.055644 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.055651 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.056141 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.056605 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.056609 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.056612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.056756 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.057331 (+ 575us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:36.057337 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:36.057498 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:12:36.057688 (+ 190us) client_negotiation.cc:240] Negotiation successful
0925 15:12:36.057993 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"spinlock_wait_cycles":4992,"thread_start_us":390,"threads_started":1}
I20260925 15:12:36.058410 27770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.040983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.4.205.254:41345)
0925 15:12:36.041169 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.041176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.041532 (+ 356us) server_negotiation.cc:407] Connection header received
0925 15:12:36.041812 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.041819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.041885 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.041938 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.043479 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.044581 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.046869 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.047638 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.047715 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.051753 (+ 4038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.051759 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.051763 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.051814 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.055489 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.056300 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.056308 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.056313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.056491 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.056888 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.056895 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.056900 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.057201 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.057351 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.057807 (+ 456us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.058014 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:12:36.059351 27767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: Bootstrap starting.
I20260925 15:12:36.059955 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:36.062960 27767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:36.063941 27767 log.cc:824] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.254@KRBTEST.COM
I20260925 15:12:36.066956 27767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: No bootstrap required, opened a new log
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:36.073992 27921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.054810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47296 (local address 127.4.205.254:41345)
0925 15:12:36.055432 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.055436 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.055454 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:12:36.055489 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.055493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.055543 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.055666 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.056959 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.058152 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.060360 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.061087 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.061227 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.066155 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.066162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.066166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.066198 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.069791 (+ 3593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.070593 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.070600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.070606 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.070756 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.072305 (+ 1549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.072309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.072311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.072605 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.072887 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.073216 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.073465 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":316,"threads_started":1}
I20260925 15:12:36.075204 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.060637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33656 (local address 127.4.205.253:33961)
0925 15:12:36.060796 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.060802 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.060881 (+ 79us) server_negotiation.cc:407] Connection header received
0925 15:12:36.061099 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.061104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.061191 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.061284 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.062320 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.063414 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.065220 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.065927 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.066051 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.069116 (+ 3065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.069121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.069124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.069154 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.072177 (+ 3023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.072927 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.072932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.072936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.073091 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.073643 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.073648 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.073651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.073981 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.074097 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.074632 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.074849 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:12:36.076035 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.060518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.253:33961 (local address 127.0.0.1:33656)
0925 15:12:36.060805 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:36.060822 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:36.060948 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:36.061449 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:36.061454 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:36.061907 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:36.062118 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.062144 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.063633 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.063639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.065052 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.065069 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.066275 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.066279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.066402 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.066927 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:36.066953 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:36.068923 (+ 1970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:36.072303 (+ 3380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.072308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.072312 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.072794 (+ 482us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.073292 (+ 498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.073298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.073303 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.073481 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.074193 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:36.074198 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:36.074346 (+ 148us) client_negotiation.cc:769] Sending connection context
0925 15:12:36.075371 (+ 1025us) client_negotiation.cc:240] Negotiation successful
0925 15:12:36.075658 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190}
I20260925 15:12:36.077086 27915 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:12:36.077478 27767 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.077934 27767 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d5d4f2612ec46b1bffe71275ecaf44f, State: Initialized, Role: FOLLOWER
I20260925 15:12:36.078451 27767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.079346 27923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.079805 27923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.080667 27767 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: configured and running, proceeding with master startup.
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.4.205.253@KRBTEST.COM
I20260925 15:12:36.091395 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.075674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.4.205.253:33961)
0925 15:12:36.075825 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.075831 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.075972 (+ 141us) server_negotiation.cc:407] Connection header received
0925 15:12:36.076503 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.076508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.076581 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.076707 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.078219 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.079379 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.080961 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.081366 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.081498 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.086272 (+ 4774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.086278 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.086291 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.086331 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.088795 (+ 2464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.089414 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.089422 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.089426 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.089608 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.089971 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.089977 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.089981 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.090254 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.090520 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.090814 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.091050 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:36.096009 4919 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.252@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:12:36.098536 27742 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47296
I20260925 15:12:36.098987 27742 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:12:36.099280 27742 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:36.102571 27742 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.104843 27742 leader_election.cc:288] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [CANDIDATE]: Term 1 election: Requested vote from peers 9b0d3d5bd1624c04956f2c4ad45acdff (127.4.205.253:33961), 8ae41802f3ab4b1cb33f7b6bddeb1fa9 (127.4.205.252:44009)
I20260925 15:12:36.105486 27920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.077924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:50990)
0925 15:12:36.078166 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:36.078183 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:36.078302 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:36.078653 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:36.078658 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:36.079115 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:36.079315 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.079326 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.089723 (+ 10397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.089731 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.091574 (+ 1843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.091591 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.094595 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.094601 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.094851 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.095435 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:36.095461 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:36.098993 (+ 3532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:36.102787 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.102794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.102798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.103130 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.103550 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.103556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.103560 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.103686 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.104443 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:36.104448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:36.104591 (+ 143us) client_negotiation.cc:769] Sending connection context
0925 15:12:36.104736 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:12:36.105052 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260925 15:12:36.106398 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.077876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50990 (local address 127.4.205.252:44009)
0925 15:12:36.078049 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.078054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.078235 (+ 181us) server_negotiation.cc:407] Connection header received
0925 15:12:36.078431 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.078437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.078505 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.078608 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.079474 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.080458 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.093707 (+ 13249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.094417 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.094557 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.099180 (+ 4623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.099189 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.099197 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.099245 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.102667 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.103274 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.103282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.103286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.103436 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.103787 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.103795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.103799 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.104261 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.104395 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.105851 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.106085 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 15:12:36.112191 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: Bootstrap starting.
I20260925 15:12:36.116287 27915 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:36.118108 27915 log.cc:824] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:36.129977 27915 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: No bootstrap required, opened a new log
I20260925 15:12:36.142194 27921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.111613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.4.205.254:41345)
0925 15:12:36.111952 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.111961 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.111983 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:12:36.112070 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.112078 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.112174 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.112297 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.114372 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.115432 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.117398 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.118132 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.118229 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.131256 (+ 13027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.131262 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.131267 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.131309 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.136354 (+ 5045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.137678 (+ 1324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.137688 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.137695 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.138210 (+ 515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.139982 (+ 1772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.139987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.139990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.140338 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.141405 (+ 1067us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.141584 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.141822 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173}
I20260925 15:12:36.151676 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.112620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51002 (local address 127.4.205.252:44009)
0925 15:12:36.117681 (+ 5061us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.117687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.117704 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:12:36.117757 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.117761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.117830 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.117922 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.122639 (+ 4717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.123749 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.126851 (+ 3102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.127534 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.127655 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.138932 (+ 11277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.138939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.138943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.138993 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.144942 (+ 5949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.147388 (+ 2446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.147402 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.147407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.147631 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.148836 (+ 1205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.148845 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.148850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.149362 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.149775 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.150847 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.151168 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4953}
I20260925 15:12:36.151546 27915 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.152014 27915 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ae41802f3ab4b1cb33f7b6bddeb1fa9, State: Initialized, Role: FOLLOWER
I20260925 15:12:36.152561 27915 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
W20260925 15:12:36.156374 27942 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:12:36.156508 27942 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:12:36.143860 27700 master_runner.cc:186] Error getting master registration for 127.4.205.253:33961: OK, instance_id { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" instance_seqno: 1790349155532828 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:12:36.158154 27700 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:12:36.159499 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.113698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.4.205.253:33961)
0925 15:12:36.115352 (+ 1654us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.115478 (+ 126us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.119408 (+ 3930us) server_negotiation.cc:407] Connection header received
0925 15:12:36.120246 (+ 838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.120291 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.120499 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.120733 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.127105 (+ 6372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.128212 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.130281 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.130932 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.131023 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.138026 (+ 7003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.138034 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.138039 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.138083 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.141747 (+ 3664us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.157321 (+ 15574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.157327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.157332 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.157494 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.157939 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.157945 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.157950 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.158352 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.158612 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.158894 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.159141 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989}
I20260925 15:12:36.162127 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.162498 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.164227 27915 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:36.166082 27890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9"
I20260925 15:12:36.166546 27890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:36.170058 27890 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5d4f2612ec46b1bffe71275ecaf44f in term 1.
I20260925 15:12:36.170624 27716 leader_election.cc:302] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d5d4f2612ec46b1bffe71275ecaf44f, 8ae41802f3ab4b1cb33f7b6bddeb1fa9; no voters:
I20260925 15:12:36.171072 27923 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:12:36.174650 27923 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [term 1 LEADER]: Becoming Leader. State: Replica: 0d5d4f2612ec46b1bffe71275ecaf44f, State: Running, Role: LEADER
I20260925 15:12:36.175240 27923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.178553 27947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5d4f2612ec46b1bffe71275ecaf44f. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.178915 27947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:36.180291 27950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:12:36.182729 27950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:12:36.191996 27890 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [term 1 FOLLOWER]: Refusing update from remote peer 0d5d4f2612ec46b1bffe71275ecaf44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:36.192791 27947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:36.212257 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5d4f2612ec46b1bffe71275ecaf44f. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.212550 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.213657 27952 mvcc.cc:205] Tried to move back new op lower bound from 7333270143753195520 to 7333270143699083264. Current Snapshot: MvccSnapshot[applied={T|T < 7333270143753195520}]
I20260925 15:12:36.221962 27953 mvcc.cc:205] Tried to move back new op lower bound from 7333270143753195520 to 7333270143699083264. Current Snapshot: MvccSnapshot[applied={T|T < 7333270143753195520}]
I20260925 15:12:36.222766 27923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.223147 27923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:36.224458 27950 catalog_manager.cc:1347] Generated new cluster ID: f90fbef1e77241e7b43378f827cc990e
I20260925 15:12:36.224575 27950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:12:36.229890 27944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.230156 27944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.256489 27841 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } attempt: 5
I20260925 15:12:36.269829 27963 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:12:36.278909 27950 catalog_manager.cc:1370] Generated new certificate authority record
Sep 25 15:12:36 dist-test-slave-64fx krb5kdc[27687](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790349155, etypes {rep=17 tkt=17 ses=17}, kudu/127.4.205.253@KRBTEST.COM for kudu/127.4.205.252@KRBTEST.COM
I20260925 15:12:36.290025 27844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.257022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.205.252:44009 (local address 127.0.0.1:51008)
0925 15:12:36.257370 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 15:12:36.257389 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:12:36.257495 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:12:36.258299 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:12:36.258304 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:12:36.258813 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:12:36.259028 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.259041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.260478 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.260483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.261945 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:12:36.261961 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.263153 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.263158 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:12:36.263389 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.279431 (+ 16042us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:12:36.279469 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:12:36.282889 (+ 3420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:12:36.287216 (+ 4327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.287222 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.287227 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.287801 (+ 574us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.288246 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:12:36.288251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:12:36.288255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:12:36.288416 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:12:36.289009 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:12:36.289014 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:12:36.289175 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 15:12:36.289320 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:12:36.289646 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260925 15:12:36.290380 27919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.257715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51008 (local address 127.4.205.252:44009)
0925 15:12:36.257877 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.257883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.257903 (+ 20us) server_negotiation.cc:407] Connection header received
0925 15:12:36.257964 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.257968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.258041 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.258131 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.259190 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.260324 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.262123 (+ 1799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.262819 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.262942 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.283073 (+ 20131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:12:36.283079 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:12:36.283084 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:12:36.283145 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:12:36.287091 (+ 3946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.287947 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.287954 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.287957 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.288103 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:12:36.288518 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:12:36.288523 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:12:36.288527 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:12:36.288886 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:12:36.289034 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.289457 (+ 423us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.289715 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:12:36.281090 27848 master_runner.cc:186] Error getting master registration for 127.4.205.253:33961: OK, instance_id { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" instance_seqno: 1790349155532828 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:12:36.292569 27848 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:12:36.293161 27950 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:12:36.296029 27841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: Bootstrap starting.
I20260925 15:12:36.300256 27841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: Neither blocks nor log segments found. Creating new log.
I20260925 15:12:36.301537 27841 log.cc:824] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: Log is configured to *not* fsync() on all Append() calls
I20260925 15:12:36.304769 27841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: No bootstrap required, opened a new log
I20260925 15:12:36.313496 27950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f: Generated new TSK 0
I20260925 15:12:36.314437 27950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:12:36.315083 27841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.315450 27841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b0d3d5bd1624c04956f2c4ad45acdff, State: Initialized, Role: FOLLOWER
I20260925 15:12:36.315943 27841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } }
I20260925 15:12:36.326083 27965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.326493 27965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.327378 27841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:12:36.331511 27816 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff"
I20260925 15:12:36.331964 27816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:12:36.336203 27816 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d5d4f2612ec46b1bffe71275ecaf44f in term 1.
I20260925 15:12:36.337069 27815 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [term 1 FOLLOWER]: Refusing update from remote peer 0d5d4f2612ec46b1bffe71275ecaf44f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:12:36.338742 27947 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:12:36.379508 27965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d5d4f2612ec46b1bffe71275ecaf44f. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.379485 27975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.379887 27965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.379889 27975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:12:36.401659 27983 catalog_manager.cc:1259] Loaded cluster ID: f90fbef1e77241e7b43378f827cc990e
I20260925 15:12:36.401759 27983 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: loading cluster ID for follower catalog manager: success
W20260925 15:12:36.408056 27921 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.394717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47312 (local address 127.4.205.254:41345)
0925 15:12:36.394878 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.394884 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.404426 (+ 9542us) server_negotiation.cc:407] Connection header received
0925 15:12:36.405972 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.405982 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.406123 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.406214 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.407705 (+ 1491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47312: BlockingRecv error: recv got EOF from 127.0.0.1:47312 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:12:36.409997 27983 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: acquiring CA information for follower catalog manager: success
W20260925 15:12:36.410418 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.401388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51022 (local address 127.4.205.252:44009)
0925 15:12:36.401583 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.401589 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.408820 (+ 7231us) server_negotiation.cc:407] Connection header received
0925 15:12:36.408977 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.408982 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.409059 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.409154 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.410163 (+ 1009us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51022: BlockingRecv error: recv got EOF from 127.0.0.1:51022 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 15:12:36.412283 27845 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.403730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.4.205.253:33961)
0925 15:12:36.403907 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.403915 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.410983 (+ 7068us) server_negotiation.cc:407] Connection header received
0925 15:12:36.411124 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.411132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.411224 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.411324 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.412076 (+ 752us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33688: BlockingRecv error: recv got EOF from 127.0.0.1:33688 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:12:36.415812 27983 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9b0d3d5bd1624c04956f2c4ad45acdff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:12:36.434631 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.427348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33696 (local address 127.4.205.253:33961)
0925 15:12:36.427502 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.427507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.427944 (+ 437us) server_negotiation.cc:407] Connection header received
0925 15:12:36.428127 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.428133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.428305 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.428469 (+ 164us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.428942 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.430210 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.432029 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.432642 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.432765 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.433306 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.433964 (+ 658us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.434114 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.434233 (+ 119us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.434319 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 15:12:36.447589 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.441297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51036 (local address 127.4.205.252:44009)
0925 15:12:36.441479 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.441485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.446055 (+ 4570us) server_negotiation.cc:407] Connection header received
0925 15:12:36.446265 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.446271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.446436 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.446542 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.447373 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51036: BlockingRecv error: recv got EOF from 127.0.0.1:51036 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 15:12:36.459838 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.451744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33708 (local address 127.4.205.253:33961)
0925 15:12:36.451934 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.451939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.454100 (+ 2161us) server_negotiation.cc:407] Connection header received
0925 15:12:36.454266 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.454271 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.454379 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.454468 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.454890 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.456065 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.457928 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.458547 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.458655 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.459097 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.459227 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.459305 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.459456 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.459528 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 15:12:36.471542 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.466603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51052 (local address 127.4.205.252:44009)
0925 15:12:36.466765 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.466771 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.470112 (+ 3341us) server_negotiation.cc:407] Connection header received
0925 15:12:36.470283 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.470289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.470399 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.470508 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.471290 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51052: BlockingRecv error: recv got EOF from 127.0.0.1:51052 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:36.483579 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.477105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33724 (local address 127.4.205.253:33961)
0925 15:12:36.477254 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.477259 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.477694 (+ 435us) server_negotiation.cc:407] Connection header received
0925 15:12:36.477825 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.477830 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.477904 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.477986 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.478404 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.479505 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.481477 (+ 1972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.482183 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.482304 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.482737 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.482869 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.482955 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.483117 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.483188 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
W20260925 15:12:36.495520 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.490002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.4.205.252:44009)
0925 15:12:36.490165 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.490170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.494056 (+ 3886us) server_negotiation.cc:407] Connection header received
0925 15:12:36.494301 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.494307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.494396 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.494499 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.495299 (+ 800us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51064: BlockingRecv error: recv got EOF from 127.0.0.1:51064 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:12:36.514227 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.507541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.4.205.253:33961)
0925 15:12:36.507753 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.507764 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.508287 (+ 523us) server_negotiation.cc:407] Connection header received
0925 15:12:36.508461 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.508467 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.508561 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.508649 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.509101 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.510303 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.512168 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.512863 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.512992 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.513415 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.513595 (+ 180us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.513697 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.513849 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.513939 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
W20260925 15:12:36.522429 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.520176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51072 (local address 127.4.205.252:44009)
0925 15:12:36.520435 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.520441 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.520954 (+ 513us) server_negotiation.cc:407] Connection header received
0925 15:12:36.521126 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.521132 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.521221 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.521325 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.522191 (+ 866us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51072: BlockingRecv error: recv got EOF from 127.0.0.1:51072 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
I20260925 15:12:36.541445 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.534516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33742 (local address 127.4.205.253:33961)
0925 15:12:36.534690 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.534696 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.535238 (+ 542us) server_negotiation.cc:407] Connection header received
0925 15:12:36.535427 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.535433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.535516 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.535618 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.536083 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.537235 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.539149 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.539895 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.539994 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.540591 (+ 597us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.540731 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.540784 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.541043 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.541130 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:12:36.573815 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.571447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.4.205.252:44009)
0925 15:12:36.571656 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.571663 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.572245 (+ 582us) server_negotiation.cc:407] Connection header received
0925 15:12:36.572462 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.572468 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.572578 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.572686 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.573598 (+ 912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51084: BlockingRecv error: recv got EOF from 127.0.0.1:51084 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 15:12:36.601045 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.593778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.4.205.253:33961)
0925 15:12:36.593975 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.593982 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.594543 (+ 561us) server_negotiation.cc:407] Connection header received
0925 15:12:36.594733 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.594739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.594841 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.594951 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.595570 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.596828 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.598814 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.599458 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.599590 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.600083 (+ 493us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.600253 (+ 170us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.600341 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.600582 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.600674 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 15:12:36.610692 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.608345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51096 (local address 127.4.205.252:44009)
0925 15:12:36.608544 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.608552 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.609097 (+ 545us) server_negotiation.cc:407] Connection header received
0925 15:12:36.609298 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.609304 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.609409 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.609507 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.610465 (+ 958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51096: BlockingRecv error: recv got EOF from 127.0.0.1:51096 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 15:12:36.653841 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.646876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33764 (local address 127.4.205.253:33961)
0925 15:12:36.647162 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.647168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.647769 (+ 601us) server_negotiation.cc:407] Connection header received
0925 15:12:36.647997 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.648003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.648097 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.648198 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.648749 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.649839 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.651749 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.652453 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.652563 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.653067 (+ 504us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.653185 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.653265 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.653429 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.653509 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 15:12:36.663486 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.660993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.4.205.252:44009)
0925 15:12:36.661192 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.661199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.662057 (+ 858us) server_negotiation.cc:407] Connection header received
0925 15:12:36.662242 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.662247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.662335 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.662430 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.663263 (+ 833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51098: BlockingRecv error: recv got EOF from 127.0.0.1:51098 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:12:36.737032 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.731264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33774 (local address 127.4.205.253:33961)
0925 15:12:36.731461 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.731470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.731786 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:12:36.731976 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.731984 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.732079 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.732194 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.732566 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.733762 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.735043 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.735753 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.735896 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.736234 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.736388 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.736495 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.736615 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.736694 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260925 15:12:36.744799 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.742882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.4.205.252:44009)
0925 15:12:36.743078 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.743084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.743319 (+ 235us) server_negotiation.cc:407] Connection header received
0925 15:12:36.743498 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.743502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.743576 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.743653 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.744550 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51112: BlockingRecv error: recv got EOF from 127.0.0.1:51112 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 15:12:36.793848 27947 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d5d4f2612ec46b1bffe71275ecaf44f" member_type: VOTER last_known_addr { host: "127.4.205.254" port: 41345 } } peers { permanent_uuid: "9b0d3d5bd1624c04956f2c4ad45acdff" member_type: VOTER last_known_addr { host: "127.4.205.253" port: 33961 } } peers { permanent_uuid: "8ae41802f3ab4b1cb33f7b6bddeb1fa9" member_type: VOTER last_known_addr { host: "127.4.205.252" port: 44009 } } }
I20260925 15:12:36.794171 27947 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d5d4f2612ec46b1bffe71275ecaf44f [sys.catalog]: This master's current role is: LEADER
I20260925 15:12:36.880858 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:36.876182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.4.205.253:33961)
0925 15:12:36.876440 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.876446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.876706 (+ 260us) server_negotiation.cc:407] Connection header received
0925 15:12:36.876845 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.876849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.876912 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.876985 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:36.877305 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.878002 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.879169 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:36.879627 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:36.879716 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:36.880051 (+ 335us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:36.880241 (+ 190us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:36.880346 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:36.880474 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:12:36.880539 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260925 15:12:36.887547 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:36.885869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51114 (local address 127.4.205.252:44009)
0925 15:12:36.886030 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:36.886034 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:36.886362 (+ 328us) server_negotiation.cc:407] Connection header received
0925 15:12:36.886489 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:36.886493 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:36.886570 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:36.886650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:36.887261 (+ 611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51114: BlockingRecv error: recv got EOF from 127.0.0.1:51114 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:12:37.151809 27845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.147154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33798 (local address 127.4.205.253:33961)
0925 15:12:37.147348 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.147353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.147669 (+ 316us) server_negotiation.cc:407] Connection header received
0925 15:12:37.147853 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.147857 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.147928 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.148015 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.148405 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.149181 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.150458 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.150835 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.150922 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.151263 (+ 341us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.151354 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.151417 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.151549 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.151609 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 15:12:37.159729 27919 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:12:37.157952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.4.205.252:44009)
0925 15:12:37.158175 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.158181 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.158367 (+ 186us) server_negotiation.cc:407] Connection header received
0925 15:12:37.158569 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.158575 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.158673 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.158778 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:12:37.159506 (+ 728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51120: BlockingRecv error: recv got EOF from 127.0.0.1:51120 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 15:12:37.274209 27963 catalog_manager.cc:1259] Loaded cluster ID: f90fbef1e77241e7b43378f827cc990e
I20260925 15:12:37.274317 27963 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: loading cluster ID for follower catalog manager: success
I20260925 15:12:37.277487 27963 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: acquiring CA information for follower catalog manager: success
I20260925 15:12:37.279305 27963 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8ae41802f3ab4b1cb33f7b6bddeb1fa9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:12:37.680083 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.675362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33806 (local address 127.4.205.253:33961)
0925 15:12:37.676070 (+ 708us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.676074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.676091 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:12:37.676152 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.676155 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.676217 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.676302 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.676675 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.677367 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.678670 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.679032 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.679118 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.679414 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.679501 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.679572 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.679686 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.679742 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":499,"threads_started":1}
I20260925 15:12:37.691236 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.686198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.4.205.252:44009)
0925 15:12:37.686608 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.686612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.686684 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:12:37.686815 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.686818 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.686959 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.687056 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.687407 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.688083 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.689271 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.689774 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.689863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.690260 (+ 397us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.690747 (+ 487us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.690863 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.690941 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.690998 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":230,"threads_started":1}
I20260925 15:12:37.710062 28105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.697242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.4.205.252:44009)
0925 15:12:37.697437 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.697444 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.704606 (+ 7162us) server_negotiation.cc:407] Connection header received
0925 15:12:37.704787 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.704793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.704897 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.704991 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.705727 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.706626 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.708443 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.708813 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.708874 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.709369 (+ 495us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.709455 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.709544 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.709750 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.709818 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:12:37.711578 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.696690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.4.205.254:41345)
0925 15:12:37.705097 (+ 8407us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.705105 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.705120 (+ 15us) server_negotiation.cc:407] Connection header received
0925 15:12:37.705222 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.705237 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.705465 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.705688 (+ 223us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.706264 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.707094 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.709239 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.709856 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.709955 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.710508 (+ 553us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.710919 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.711030 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.711237 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.711324 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8288,"thread_start_us":8185,"threads_started":1}
I20260925 15:12:37.711692 28099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:12:37.696960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33822 (local address 127.4.205.253:33961)
0925 15:12:37.702367 (+ 5407us) server_negotiation.cc:206] Beginning negotiation
0925 15:12:37.702376 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:12:37.705574 (+ 3198us) server_negotiation.cc:407] Connection header received
0925 15:12:37.705818 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:12:37.705823 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:12:37.705903 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:12:37.705985 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:12:37.706579 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.707580 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.709617 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:12:37.710195 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:12:37.710295 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:12:37.710798 (+ 503us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:12:37.710893 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:12:37.710987 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:12:37.711380 (+ 393us) server_negotiation.cc:299] Negotiation successful
0925 15:12:37.711457 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5178}
I20260925 15:12:37.716006 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27700
I20260925 15:12:37.717068 27762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:37.822918 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27700
W20260925 15:12:37.842653 27866 connection.cc:535] server connection from 127.0.0.1:50988 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:12:37.843056 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27774
I20260925 15:12:37.844038 27836 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:37.947445 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27774
I20260925 15:12:37.967084 4919 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu pid 27848
I20260925 15:12:37.968155 27910 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 15:12:38.070215 4919 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskSXSwaN/build/asan/bin/kudu with pid 27848
2026-09-25T15:12:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4690 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9113 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (349127 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST